2023-12-16T07:17:27.722 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-16T07:17:27.722 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi --verbose --archive /home/teuthworker/archive/yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi/7493743 --description orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} -- /home/teuthworker/archive/yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi/7493743/orig.config.yaml 2023-12-16T07:17:27.756 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-16T07:17:27.836 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi/7493743 branch: main description: orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} email: yweinste@redhat.com first_in_suite: false job_id: '7493743' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 6d933094a156ec7587742e5aa25fba5ba27ec678 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 6d933094a156ec7587742e5aa25fba5ba27ec678 workunit: branch: main sha1: 6d933094a156ec7587742e5aa25fba5ba27ec678 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 2407 sha1: 6d933094a156ec7587742e5aa25fba5ba27ec678 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 6d933094a156ec7587742e5aa25fba5ba27ec678 targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-16_00:10:43 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-16T07:17:27.837 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa; will attempt to use it 2023-12-16T07:17:27.837 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks 2023-12-16T07:17:27.837 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-16T07:17:27.839 INFO:teuthology.task.internal:Checking packages... 2023-12-16T07:17:27.865 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '6d933094a156ec7587742e5aa25fba5ba27ec678' 2023-12-16T07:17:27.866 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-16T07:17:27.866 INFO:teuthology.packaging:ref: None 2023-12-16T07:17:27.866 INFO:teuthology.packaging:tag: None 2023-12-16T07:17:27.866 INFO:teuthology.packaging:branch: main 2023-12-16T07:17:27.866 INFO:teuthology.packaging:sha1: 6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T07:17:27.866 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=main 2023-12-16T07:17:28.307 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-196-g85fe3c65-1focal 2023-12-16T07:17:28.308 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-16T07:17:28.315 INFO:teuthology.task.internal:no buildpackages task found 2023-12-16T07:17:28.315 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-16T07:17:28.321 INFO:teuthology.task.internal:Saving configuration 2023-12-16T07:17:28.333 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-16T07:17:28.339 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T07:17:28.363 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi/7493743', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 07:11:03.918700', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-12-16T07:17:28.384 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi/7493743', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 07:11:03.920678', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-12-16T07:17:28.385 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-16T07:17:28.391 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2023-12-16T07:17:28.391 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2023-12-16T07:17:28.391 INFO:teuthology.run_tasks:Running task console_log... 2023-12-16T07:17:28.470 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff585e23e50>, signals=[15]) 2023-12-16T07:17:28.470 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-16T07:17:28.477 INFO:teuthology.task.internal:Opening connections... 2023-12-16T07:17:28.477 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2023-12-16T07:17:28.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T07:17:28.528 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2023-12-16T07:17:28.529 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T07:17:28.574 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-16T07:17:28.582 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2023-12-16T07:17:28.612 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2023-12-16T07:17:28.612 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2023-12-16T07:17:28.661 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="Ubuntu" 2023-12-16T07:17:28.661 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-12-16T07:17:28.661 INFO:teuthology.orchestra.run.smithi007.stdout:ID=ubuntu 2023-12-16T07:17:28.661 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE=debian 2023-12-16T07:17:28.661 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-12-16T07:17:28.661 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="20.04" 2023-12-16T07:17:28.661 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-16T07:17:28.661 INFO:teuthology.orchestra.run.smithi007.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-16T07:17:28.661 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-16T07:17:28.662 INFO:teuthology.orchestra.run.smithi007.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-16T07:17:28.662 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_CODENAME=focal 2023-12-16T07:17:28.662 INFO:teuthology.orchestra.run.smithi007.stdout:UBUNTU_CODENAME=focal 2023-12-16T07:17:28.662 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2023-12-16T07:17:28.686 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2023-12-16T07:17:28.694 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2023-12-16T07:17:28.694 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2023-12-16T07:17:28.742 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="Ubuntu" 2023-12-16T07:17:28.743 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-12-16T07:17:28.743 INFO:teuthology.orchestra.run.smithi169.stdout:ID=ubuntu 2023-12-16T07:17:28.743 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE=debian 2023-12-16T07:17:28.743 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-12-16T07:17:28.743 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="20.04" 2023-12-16T07:17:28.743 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-16T07:17:28.743 INFO:teuthology.orchestra.run.smithi169.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-16T07:17:28.743 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-16T07:17:28.743 INFO:teuthology.orchestra.run.smithi169.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-16T07:17:28.743 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_CODENAME=focal 2023-12-16T07:17:28.743 INFO:teuthology.orchestra.run.smithi169.stdout:UBUNTU_CODENAME=focal 2023-12-16T07:17:28.744 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2023-12-16T07:17:28.769 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-16T07:17:28.779 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-16T07:17:28.841 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-16T07:17:28.841 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2023-12-16T07:17:28.843 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2023-12-16T07:17:28.849 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-16T07:17:28.855 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-16T07:17:28.855 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2023-12-16T07:17:28.858 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2023-12-16T07:17:28.874 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-16T07:17:28.927 INFO:teuthology.run_tasks:Running task kernel... 2023-12-16T07:17:28.942 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-16T07:17:28.942 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-16T07:17:28.942 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-16T07:17:28.943 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-16T07:17:28.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T07:17:28.948 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-12-16T07:17:28.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T07:17:28.949 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2023-12-16T07:17:28.955 INFO:teuthology.orchestra.run.smithi007.stdout:5.4.0-124-generic 2023-12-16T07:17:28.955 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.4.0-124-generic 2023-12-16T07:17:28.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get clean 2023-12-16T07:17:28.957 INFO:teuthology.orchestra.run.smithi169.stdout:5.4.0-124-generic 2023-12-16T07:17:28.957 INFO:teuthology.task.kernel:Running kernel on smithi169: 5.4.0-124-generic 2023-12-16T07:17:28.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get clean 2023-12-16T07:17:29.086 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2023-12-16T07:17:29.127 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get update 2023-12-16T07:17:29.289 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-16T07:17:29.290 INFO:teuthology.orchestra.run.smithi169.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-12-16T07:17:29.290 INFO:teuthology.orchestra.run.smithi169.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-12-16T07:17:29.375 INFO:teuthology.orchestra.run.smithi169.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-12-16T07:17:29.401 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-16T07:17:29.401 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-12-16T07:17:29.450 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-12-16T07:17:29.691 INFO:teuthology.orchestra.run.smithi169.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [693 kB] 2023-12-16T07:17:29.794 INFO:teuthology.orchestra.run.smithi169.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,641 kB] 2023-12-16T07:17:29.806 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-12-16T07:17:29.834 INFO:teuthology.orchestra.run.smithi169.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,023 kB] 2023-12-16T07:17:29.836 INFO:teuthology.orchestra.run.smithi169.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [405 kB] 2023-12-16T07:17:29.840 INFO:teuthology.orchestra.run.smithi169.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-12-16T07:17:29.840 INFO:teuthology.orchestra.run.smithi169.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,451 kB] 2023-12-16T07:17:29.859 INFO:teuthology.orchestra.run.smithi169.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [34.7 kB] 2023-12-16T07:17:29.860 INFO:teuthology.orchestra.run.smithi169.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [342 kB] 2023-12-16T07:17:29.862 INFO:teuthology.orchestra.run.smithi169.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2023-12-16T07:17:29.863 INFO:teuthology.orchestra.run.smithi169.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [916 kB] 2023-12-16T07:17:29.870 INFO:teuthology.orchestra.run.smithi169.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [635 kB] 2023-12-16T07:17:29.874 INFO:teuthology.orchestra.run.smithi169.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [192 kB] 2023-12-16T07:17:29.875 INFO:teuthology.orchestra.run.smithi169.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2023-12-16T07:17:29.876 INFO:teuthology.orchestra.run.smithi169.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-12-16T07:17:29.876 INFO:teuthology.orchestra.run.smithi169.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,188 B] 2023-12-16T07:17:29.876 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [693 kB] 2023-12-16T07:17:29.888 INFO:teuthology.orchestra.run.smithi169.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-12-16T07:17:29.888 INFO:teuthology.orchestra.run.smithi169.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-12-16T07:17:29.953 INFO:teuthology.orchestra.run.smithi169.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [919 kB] 2023-12-16T07:17:29.963 INFO:teuthology.orchestra.run.smithi169.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [488 kB] 2023-12-16T07:17:29.967 INFO:teuthology.orchestra.run.smithi169.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2023-12-16T07:17:29.967 INFO:teuthology.orchestra.run.smithi169.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,569 kB] 2023-12-16T07:17:29.995 INFO:teuthology.orchestra.run.smithi169.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [36.0 kB] 2023-12-16T07:17:29.995 INFO:teuthology.orchestra.run.smithi169.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [359 kB] 2023-12-16T07:17:30.001 INFO:teuthology.orchestra.run.smithi169.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2023-12-16T07:17:30.001 INFO:teuthology.orchestra.run.smithi169.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,142 kB] 2023-12-16T07:17:30.012 INFO:teuthology.orchestra.run.smithi169.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [762 kB] 2023-12-16T07:17:30.020 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,023 kB] 2023-12-16T07:17:30.020 INFO:teuthology.orchestra.run.smithi169.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [273 kB] 2023-12-16T07:17:30.022 INFO:teuthology.orchestra.run.smithi169.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2023-12-16T07:17:30.023 INFO:teuthology.orchestra.run.smithi169.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2023-12-16T07:17:30.023 INFO:teuthology.orchestra.run.smithi169.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-12-16T07:17:30.023 INFO:teuthology.orchestra.run.smithi169.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-12-16T07:17:30.024 INFO:teuthology.orchestra.run.smithi169.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-12-16T07:17:30.043 INFO:teuthology.orchestra.run.smithi169.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-12-16T07:17:30.044 INFO:teuthology.orchestra.run.smithi169.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-12-16T07:17:30.044 INFO:teuthology.orchestra.run.smithi169.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-12-16T07:17:30.045 INFO:teuthology.orchestra.run.smithi169.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-12-16T07:17:30.063 INFO:teuthology.orchestra.run.smithi169.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-12-16T07:17:30.063 INFO:teuthology.orchestra.run.smithi169.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-12-16T07:17:30.245 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [2,641 kB] 2023-12-16T07:17:30.435 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [405 kB] 2023-12-16T07:17:30.450 INFO:teuthology.orchestra.run.smithi007.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [13.2 kB] 2023-12-16T07:17:30.450 INFO:teuthology.orchestra.run.smithi007.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [2,451 kB] 2023-12-16T07:17:30.493 INFO:teuthology.orchestra.run.smithi007.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [919 kB] 2023-12-16T07:17:30.511 INFO:teuthology.orchestra.run.smithi007.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [488 kB] 2023-12-16T07:17:30.519 INFO:teuthology.orchestra.run.smithi007.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [34.7 kB] 2023-12-16T07:17:30.519 INFO:teuthology.orchestra.run.smithi007.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [342 kB] 2023-12-16T07:17:30.520 INFO:teuthology.orchestra.run.smithi007.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.2 kB] 2023-12-16T07:17:30.520 INFO:teuthology.orchestra.run.smithi007.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [36.0 kB] 2023-12-16T07:17:30.521 INFO:teuthology.orchestra.run.smithi007.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [2,569 kB] 2023-12-16T07:17:30.531 INFO:teuthology.orchestra.run.smithi007.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [552 B] 2023-12-16T07:17:30.531 INFO:teuthology.orchestra.run.smithi007.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [916 kB] 2023-12-16T07:17:30.553 INFO:teuthology.orchestra.run.smithi007.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [635 kB] 2023-12-16T07:17:30.604 INFO:teuthology.orchestra.run.smithi007.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [359 kB] 2023-12-16T07:17:30.609 INFO:teuthology.orchestra.run.smithi007.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [552 B] 2023-12-16T07:17:30.747 INFO:teuthology.orchestra.run.smithi007.stdout:Get:23 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [192 kB] 2023-12-16T07:17:30.774 INFO:teuthology.orchestra.run.smithi007.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [762 kB] 2023-12-16T07:17:31.178 INFO:teuthology.orchestra.run.smithi007.stdout:Get:25 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [19.2 kB] 2023-12-16T07:17:31.184 INFO:teuthology.orchestra.run.smithi007.stdout:Get:26 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [23.6 kB] 2023-12-16T07:17:31.191 INFO:teuthology.orchestra.run.smithi007.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,188 B] 2023-12-16T07:17:31.193 INFO:teuthology.orchestra.run.smithi007.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,504 B] 2023-12-16T07:17:31.195 INFO:teuthology.orchestra.run.smithi007.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [548 B] 2023-12-16T07:17:31.364 INFO:teuthology.orchestra.run.smithi007.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,142 kB] 2023-12-16T07:17:31.476 INFO:teuthology.orchestra.run.smithi007.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [273 kB] 2023-12-16T07:17:31.496 INFO:teuthology.orchestra.run.smithi007.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [25.7 kB] 2023-12-16T07:17:31.498 INFO:teuthology.orchestra.run.smithi007.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2023-12-16T07:17:31.500 INFO:teuthology.orchestra.run.smithi007.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.8 kB] 2023-12-16T07:17:31.503 INFO:teuthology.orchestra.run.smithi007.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,484 B] 2023-12-16T07:17:31.504 INFO:teuthology.orchestra.run.smithi007.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [620 B] 2023-12-16T07:17:31.504 INFO:teuthology.orchestra.run.smithi007.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2023-12-16T07:17:31.507 INFO:teuthology.orchestra.run.smithi007.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2023-12-16T07:17:31.509 INFO:teuthology.orchestra.run.smithi007.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2023-12-16T07:17:31.677 INFO:teuthology.orchestra.run.smithi007.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-12-16T07:17:31.841 INFO:teuthology.orchestra.run.smithi007.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2023-12-16T07:17:31.892 INFO:teuthology.orchestra.run.smithi007.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-12-16T07:17:33.777 INFO:teuthology.orchestra.run.smithi169.stdout:Fetched 18.5 MB in 3s (7,045 kB/s) 2023-12-16T07:17:34.784 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 18.5 MB in 3s (5,468 kB/s) 2023-12-16T07:17:35.158 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2023-12-16T07:17:35.181 DEBUG:teuthology.orchestra.run.smithi169:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-16T07:17:35.254 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2023-12-16T07:17:35.457 INFO:teuthology.orchestra.run.smithi169.stdout:Building dependency tree... 2023-12-16T07:17:35.458 INFO:teuthology.orchestra.run.smithi169.stdout:Reading state information... 2023-12-16T07:17:35.639 INFO:teuthology.orchestra.run.smithi169.stdout:The following additional packages will be installed: 2023-12-16T07:17:35.640 INFO:teuthology.orchestra.run.smithi169.stdout: linux-generic linux-headers-5.4.0-169 linux-headers-5.4.0-169-generic 2023-12-16T07:17:35.640 INFO:teuthology.orchestra.run.smithi169.stdout: linux-headers-generic linux-image-5.4.0-169-generic 2023-12-16T07:17:35.640 INFO:teuthology.orchestra.run.smithi169.stdout: linux-modules-5.4.0-169-generic linux-modules-extra-5.4.0-169-generic 2023-12-16T07:17:35.641 INFO:teuthology.orchestra.run.smithi169.stdout:Suggested packages: 2023-12-16T07:17:35.641 INFO:teuthology.orchestra.run.smithi169.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-12-16T07:17:35.641 INFO:teuthology.orchestra.run.smithi169.stdout:Recommended packages: 2023-12-16T07:17:35.641 INFO:teuthology.orchestra.run.smithi169.stdout: thermald 2023-12-16T07:17:35.697 INFO:teuthology.orchestra.run.smithi169.stdout:The following NEW packages will be installed: 2023-12-16T07:17:35.698 INFO:teuthology.orchestra.run.smithi169.stdout: linux-headers-5.4.0-169 linux-headers-5.4.0-169-generic 2023-12-16T07:17:35.698 INFO:teuthology.orchestra.run.smithi169.stdout: linux-image-5.4.0-169-generic linux-modules-5.4.0-169-generic 2023-12-16T07:17:35.698 INFO:teuthology.orchestra.run.smithi169.stdout: linux-modules-extra-5.4.0-169-generic 2023-12-16T07:17:35.699 INFO:teuthology.orchestra.run.smithi169.stdout:The following packages will be upgraded: 2023-12-16T07:17:35.700 INFO:teuthology.orchestra.run.smithi169.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-16T07:17:35.882 INFO:teuthology.orchestra.run.smithi169.stdout:3 upgraded, 5 newly installed, 0 to remove and 280 not upgraded. 2023-12-16T07:17:35.882 INFO:teuthology.orchestra.run.smithi169.stdout:Need to get 77.1 MB of archives. 2023-12-16T07:17:35.882 INFO:teuthology.orchestra.run.smithi169.stdout:After this operation, 380 MB of additional disk space will be used. 2023-12-16T07:17:35.882 INFO:teuthology.orchestra.run.smithi169.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-169-generic amd64 5.4.0-169.187 [15.0 MB] 2023-12-16T07:17:36.404 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2023-12-16T07:17:36.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-16T07:17:36.510 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2023-12-16T07:17:36.720 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2023-12-16T07:17:36.720 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2023-12-16T07:17:36.927 INFO:teuthology.orchestra.run.smithi007.stdout:The following additional packages will be installed: 2023-12-16T07:17:36.927 INFO:teuthology.orchestra.run.smithi007.stdout: linux-generic linux-headers-5.4.0-169 linux-headers-5.4.0-169-generic 2023-12-16T07:17:36.927 INFO:teuthology.orchestra.run.smithi007.stdout: linux-headers-generic linux-image-5.4.0-169-generic 2023-12-16T07:17:36.928 INFO:teuthology.orchestra.run.smithi007.stdout: linux-modules-5.4.0-169-generic linux-modules-extra-5.4.0-169-generic 2023-12-16T07:17:36.929 INFO:teuthology.orchestra.run.smithi007.stdout:Suggested packages: 2023-12-16T07:17:36.929 INFO:teuthology.orchestra.run.smithi007.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-12-16T07:17:36.929 INFO:teuthology.orchestra.run.smithi007.stdout:Recommended packages: 2023-12-16T07:17:36.929 INFO:teuthology.orchestra.run.smithi007.stdout: thermald 2023-12-16T07:17:36.992 INFO:teuthology.orchestra.run.smithi007.stdout:The following NEW packages will be installed: 2023-12-16T07:17:36.992 INFO:teuthology.orchestra.run.smithi007.stdout: linux-headers-5.4.0-169 linux-headers-5.4.0-169-generic 2023-12-16T07:17:36.993 INFO:teuthology.orchestra.run.smithi007.stdout: linux-image-5.4.0-169-generic linux-modules-5.4.0-169-generic 2023-12-16T07:17:36.993 INFO:teuthology.orchestra.run.smithi007.stdout: linux-modules-extra-5.4.0-169-generic 2023-12-16T07:17:36.994 INFO:teuthology.orchestra.run.smithi007.stdout:The following packages will be upgraded: 2023-12-16T07:17:36.995 INFO:teuthology.orchestra.run.smithi007.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-16T07:17:37.056 INFO:teuthology.orchestra.run.smithi169.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-169-generic amd64 5.4.0-169.187 [10.5 MB] 2023-12-16T07:17:37.113 INFO:teuthology.orchestra.run.smithi007.stdout:3 upgraded, 5 newly installed, 0 to remove and 280 not upgraded. 2023-12-16T07:17:37.113 INFO:teuthology.orchestra.run.smithi007.stdout:Need to get 77.1 MB of archives. 2023-12-16T07:17:37.113 INFO:teuthology.orchestra.run.smithi007.stdout:After this operation, 380 MB of additional disk space will be used. 2023-12-16T07:17:37.113 INFO:teuthology.orchestra.run.smithi007.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-169-generic amd64 5.4.0-169.187 [15.0 MB] 2023-12-16T07:17:37.384 INFO:teuthology.orchestra.run.smithi169.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-169-generic amd64 5.4.0-169.187 [39.2 MB] 2023-12-16T07:17:38.057 INFO:teuthology.orchestra.run.smithi007.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-169-generic amd64 5.4.0-169.187 [10.5 MB] 2023-12-16T07:17:38.244 INFO:teuthology.orchestra.run.smithi007.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-169-generic amd64 5.4.0-169.187 [39.2 MB] 2023-12-16T07:17:38.644 INFO:teuthology.orchestra.run.smithi169.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.169.167 [1,900 B] 2023-12-16T07:17:38.644 INFO:teuthology.orchestra.run.smithi169.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.169.167 [2,520 B] 2023-12-16T07:17:38.644 INFO:teuthology.orchestra.run.smithi169.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-169 all 5.4.0-169.187 [11.0 MB] 2023-12-16T07:17:38.955 INFO:teuthology.orchestra.run.smithi169.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-169-generic amd64 5.4.0-169.187 [1,374 kB] 2023-12-16T07:17:38.999 INFO:teuthology.orchestra.run.smithi169.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.169.167 [2,392 B] 2023-12-16T07:17:39.200 INFO:teuthology.orchestra.run.smithi007.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.169.167 [1,900 B] 2023-12-16T07:17:39.201 INFO:teuthology.orchestra.run.smithi007.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.169.167 [2,520 B] 2023-12-16T07:17:39.201 INFO:teuthology.orchestra.run.smithi007.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-169 all 5.4.0-169.187 [11.0 MB] 2023-12-16T07:17:39.384 INFO:teuthology.orchestra.run.smithi007.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-169-generic amd64 5.4.0-169.187 [1,374 kB] 2023-12-16T07:17:39.409 INFO:teuthology.orchestra.run.smithi007.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.169.167 [2,392 B] 2023-12-16T07:17:39.688 INFO:teuthology.orchestra.run.smithi169.stdout:Fetched 77.1 MB in 3s (23.4 MB/s) 2023-12-16T07:17:39.878 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-modules-5.4.0-169-generic. 2023-12-16T07:17:40.196 INFO:teuthology.orchestra.run.smithi007.stdout:Fetched 77.1 MB in 2s (32.0 MB/s) 2023-12-16T07:17:40.397 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-modules-5.4.0-169-generic. 2023-12-16T07:17:41.642 INFO:teuthology.orchestra.run.smithi169.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-12-16T07:17:41.647 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../0-linux-modules-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-16T07:17:41.733 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-modules-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:17:42.128 INFO:teuthology.orchestra.run.smithi007.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-12-16T07:17:42.133 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../0-linux-modules-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-16T07:17:42.280 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-modules-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:17:43.700 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-image-5.4.0-169-generic. 2023-12-16T07:17:43.714 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../1-linux-image-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-16T07:17:43.806 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:17:44.381 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-image-5.4.0-169-generic. 2023-12-16T07:17:44.396 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../1-linux-image-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-16T07:17:44.525 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:17:44.904 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-169-generic. 2023-12-16T07:17:44.917 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-16T07:17:44.953 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-modules-extra-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:17:45.726 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-169-generic. 2023-12-16T07:17:45.742 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-16T07:17:45.817 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-modules-extra-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:17:49.559 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../3-linux-generic_5.4.0.169.167_amd64.deb ... 2023-12-16T07:17:49.703 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2023-12-16T07:17:50.053 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.169.167_amd64.deb ... 2023-12-16T07:17:50.206 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-image-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2023-12-16T07:17:50.526 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-headers-5.4.0-169. 2023-12-16T07:17:50.541 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../5-linux-headers-5.4.0-169_5.4.0-169.187_all.deb ... 2023-12-16T07:17:50.609 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-headers-5.4.0-169 (5.4.0-169.187) ... 2023-12-16T07:17:50.906 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../3-linux-generic_5.4.0.169.167_amd64.deb ... 2023-12-16T07:17:51.325 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2023-12-16T07:17:51.684 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.169.167_amd64.deb ... 2023-12-16T07:17:51.812 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-image-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2023-12-16T07:17:52.157 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-headers-5.4.0-169. 2023-12-16T07:17:52.170 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../5-linux-headers-5.4.0-169_5.4.0-169.187_all.deb ... 2023-12-16T07:17:52.231 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-headers-5.4.0-169 (5.4.0-169.187) ... 2023-12-16T07:17:55.866 INFO:teuthology.orchestra.run.smithi169.stdout:Selecting previously unselected package linux-headers-5.4.0-169-generic. 2023-12-16T07:17:55.889 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../6-linux-headers-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-16T07:17:55.933 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-headers-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:17:57.811 INFO:teuthology.orchestra.run.smithi007.stdout:Selecting previously unselected package linux-headers-5.4.0-169-generic. 2023-12-16T07:17:57.833 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../6-linux-headers-5.4.0-169-generic_5.4.0-169.187_amd64.deb ... 2023-12-16T07:17:57.878 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-headers-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:17:58.613 INFO:teuthology.orchestra.run.smithi169.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.169.167_amd64.deb ... 2023-12-16T07:17:58.773 INFO:teuthology.orchestra.run.smithi169.stdout:Unpacking linux-headers-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2023-12-16T07:17:59.099 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-headers-5.4.0-169 (5.4.0-169.187) ... 2023-12-16T07:17:59.203 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-headers-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:17:59.304 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-modules-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:18:00.658 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:18:00.828 INFO:teuthology.orchestra.run.smithi007.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.169.167_amd64.deb ... 2023-12-16T07:18:01.005 INFO:teuthology.orchestra.run.smithi007.stdout:Unpacking linux-headers-generic (5.4.0.169.167) over (5.4.0.124.125) ... 2023-12-16T07:18:01.390 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-headers-5.4.0-169 (5.4.0-169.187) ... 2023-12-16T07:18:01.817 INFO:teuthology.orchestra.run.smithi169.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-169-generic 2023-12-16T07:18:01.817 INFO:teuthology.orchestra.run.smithi169.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-169-generic 2023-12-16T07:18:01.952 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-headers-generic (5.4.0.169.167) ... 2023-12-16T07:18:02.087 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-modules-extra-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:18:02.360 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-headers-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:18:02.711 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-modules-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:18:03.263 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-image-generic (5.4.0.169.167) ... 2023-12-16T07:18:03.389 INFO:teuthology.orchestra.run.smithi169.stdout:Setting up linux-generic (5.4.0.169.167) ... 2023-12-16T07:18:03.507 INFO:teuthology.orchestra.run.smithi169.stdout:Processing triggers for linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:18:03.583 INFO:teuthology.orchestra.run.smithi169.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-16T07:18:03.583 INFO:teuthology.orchestra.run.smithi169.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-169-generic 2023-12-16T07:18:04.000 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:18:05.188 INFO:teuthology.orchestra.run.smithi007.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-169-generic 2023-12-16T07:18:05.189 INFO:teuthology.orchestra.run.smithi007.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-169-generic 2023-12-16T07:18:05.356 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-headers-generic (5.4.0.169.167) ... 2023-12-16T07:18:05.482 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-modules-extra-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:18:06.751 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-image-generic (5.4.0.169.167) ... 2023-12-16T07:18:06.902 INFO:teuthology.orchestra.run.smithi007.stdout:Setting up linux-generic (5.4.0.169.167) ... 2023-12-16T07:18:07.028 INFO:teuthology.orchestra.run.smithi007.stdout:Processing triggers for linux-image-5.4.0-169-generic (5.4.0-169.187) ... 2023-12-16T07:18:07.116 INFO:teuthology.orchestra.run.smithi007.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-16T07:18:07.116 INFO:teuthology.orchestra.run.smithi007.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-169-generic 2023-12-16T07:18:25.454 INFO:teuthology.orchestra.run.smithi169.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-16T07:18:25.454 INFO:teuthology.orchestra.run.smithi169.stdout:Sourcing file `/etc/default/grub' 2023-12-16T07:18:25.465 INFO:teuthology.orchestra.run.smithi169.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-16T07:18:25.483 INFO:teuthology.orchestra.run.smithi169.stdout:Generating grub configuration file ... 2023-12-16T07:18:25.813 INFO:teuthology.orchestra.run.smithi169.stdout:Found linux image: /boot/vmlinuz-5.4.0-169-generic 2023-12-16T07:18:25.835 INFO:teuthology.orchestra.run.smithi169.stdout:Found initrd image: /boot/initrd.img-5.4.0-169-generic 2023-12-16T07:18:26.213 INFO:teuthology.orchestra.run.smithi169.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-16T07:18:26.233 INFO:teuthology.orchestra.run.smithi169.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-16T07:18:28.213 INFO:teuthology.orchestra.run.smithi169.stdout:done 2023-12-16T07:18:28.616 DEBUG:teuthology.orchestra.run.smithi169:> dpkg -s linux-image-generic 2023-12-16T07:18:28.670 INFO:teuthology.orchestra.run.smithi169.stdout:Package: linux-image-generic 2023-12-16T07:18:28.670 INFO:teuthology.orchestra.run.smithi169.stdout:Status: install ok installed 2023-12-16T07:18:28.670 INFO:teuthology.orchestra.run.smithi169.stdout:Priority: optional 2023-12-16T07:18:28.671 INFO:teuthology.orchestra.run.smithi169.stdout:Section: kernel 2023-12-16T07:18:28.671 INFO:teuthology.orchestra.run.smithi169.stdout:Installed-Size: 20 2023-12-16T07:18:28.671 INFO:teuthology.orchestra.run.smithi169.stdout:Maintainer: Ubuntu Kernel Team 2023-12-16T07:18:28.671 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: amd64 2023-12-16T07:18:28.671 INFO:teuthology.orchestra.run.smithi169.stdout:Source: linux-meta 2023-12-16T07:18:28.671 INFO:teuthology.orchestra.run.smithi169.stdout:Version: 5.4.0.169.167 2023-12-16T07:18:28.671 INFO:teuthology.orchestra.run.smithi169.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-12-16T07:18:28.671 INFO:teuthology.orchestra.run.smithi169.stdout:Depends: linux-image-5.4.0-169-generic, linux-modules-extra-5.4.0-169-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-16T07:18:28.671 INFO:teuthology.orchestra.run.smithi169.stdout:Recommends: thermald 2023-12-16T07:18:28.671 INFO:teuthology.orchestra.run.smithi169.stdout:Description: Generic Linux kernel image 2023-12-16T07:18:28.671 INFO:teuthology.orchestra.run.smithi169.stdout: This package will always depend on the latest generic kernel image 2023-12-16T07:18:28.671 INFO:teuthology.orchestra.run.smithi169.stdout: available. 2023-12-16T07:18:28.672 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-169-generic 2023-12-16T07:18:28.672 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-16T07:18:28.672 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-16T07:18:28.673 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-16T07:18:28.673 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2023-12-16T07:18:28.673 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get clean 2023-12-16T07:18:28.819 DEBUG:teuthology.orchestra.run.smithi169:> sudo apt-get update 2023-12-16T07:18:28.962 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-16T07:18:28.973 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-12-16T07:18:28.993 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-12-16T07:18:29.097 INFO:teuthology.orchestra.run.smithi169.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-12-16T07:18:29.606 INFO:teuthology.orchestra.run.smithi007.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-16T07:18:29.606 INFO:teuthology.orchestra.run.smithi007.stdout:Sourcing file `/etc/default/grub' 2023-12-16T07:18:29.616 INFO:teuthology.orchestra.run.smithi007.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-16T07:18:29.640 INFO:teuthology.orchestra.run.smithi007.stdout:Generating grub configuration file ... 2023-12-16T07:18:29.972 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.4.0-169-generic 2023-12-16T07:18:29.996 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.4.0-169-generic 2023-12-16T07:18:30.443 INFO:teuthology.orchestra.run.smithi007.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-16T07:18:30.457 INFO:teuthology.orchestra.run.smithi007.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-16T07:18:30.723 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2023-12-16T07:18:30.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-16T07:18:30.813 INFO:teuthology.orchestra.run.smithi169.stdout:Reading package lists... 2023-12-16T07:18:31.017 INFO:teuthology.orchestra.run.smithi169.stdout:Building dependency tree... 2023-12-16T07:18:31.017 INFO:teuthology.orchestra.run.smithi169.stdout:Reading state information... 2023-12-16T07:18:31.237 INFO:teuthology.orchestra.run.smithi169.stdout:linux-image-generic is already the newest version (5.4.0.169.167). 2023-12-16T07:18:31.237 INFO:teuthology.orchestra.run.smithi169.stdout:0 upgraded, 0 newly installed, 0 to remove and 280 not upgraded. 2023-12-16T07:18:31.239 DEBUG:teuthology.orchestra.run.smithi169:> dpkg -s linux-image-generic 2023-12-16T07:18:31.260 INFO:teuthology.orchestra.run.smithi169.stdout:Package: linux-image-generic 2023-12-16T07:18:31.260 INFO:teuthology.orchestra.run.smithi169.stdout:Status: install ok installed 2023-12-16T07:18:31.260 INFO:teuthology.orchestra.run.smithi169.stdout:Priority: optional 2023-12-16T07:18:31.260 INFO:teuthology.orchestra.run.smithi169.stdout:Section: kernel 2023-12-16T07:18:31.260 INFO:teuthology.orchestra.run.smithi169.stdout:Installed-Size: 20 2023-12-16T07:18:31.260 INFO:teuthology.orchestra.run.smithi169.stdout:Maintainer: Ubuntu Kernel Team 2023-12-16T07:18:31.261 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: amd64 2023-12-16T07:18:31.261 INFO:teuthology.orchestra.run.smithi169.stdout:Source: linux-meta 2023-12-16T07:18:31.261 INFO:teuthology.orchestra.run.smithi169.stdout:Version: 5.4.0.169.167 2023-12-16T07:18:31.261 INFO:teuthology.orchestra.run.smithi169.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-12-16T07:18:31.261 INFO:teuthology.orchestra.run.smithi169.stdout:Depends: linux-image-5.4.0-169-generic, linux-modules-extra-5.4.0-169-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-16T07:18:31.261 INFO:teuthology.orchestra.run.smithi169.stdout:Recommends: thermald 2023-12-16T07:18:31.261 INFO:teuthology.orchestra.run.smithi169.stdout:Description: Generic Linux kernel image 2023-12-16T07:18:31.261 INFO:teuthology.orchestra.run.smithi169.stdout: This package will always depend on the latest generic kernel image 2023-12-16T07:18:31.262 INFO:teuthology.orchestra.run.smithi169.stdout: available. 2023-12-16T07:18:31.262 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2023-12-16T07:18:31.266 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.7mWER2yMBm 2023-12-16T07:18:31.267 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub/grub.cfg /tmp/tmp.7mWER2yMBm 2023-12-16T07:18:31.325 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.7mWER2yMBm 2023-12-16T07:18:31.365 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.7mWER2yMBm is 9KB 2023-12-16T07:18:31.379 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.7mWER2yMBm 2023-12-16T07:18:31.385 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-16T07:18:31.444 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T07:18:31.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-16T07:18:31.444 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-16T07:18:31.475 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-169-generic 2023-12-16T07:18:31.475 DEBUG:teuthology.orchestra.run.smithi169:> sudo update-grub 2023-12-16T07:18:32.352 INFO:teuthology.orchestra.run.smithi007.stdout:done 2023-12-16T07:18:32.794 INFO:teuthology.orchestra.run.smithi169.stderr:Sourcing file `/etc/default/grub' 2023-12-16T07:18:32.801 DEBUG:teuthology.orchestra.run.smithi007:> dpkg -s linux-image-generic 2023-12-16T07:18:32.804 INFO:teuthology.orchestra.run.smithi169.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-16T07:18:32.816 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2023-12-16T07:18:32.846 INFO:teuthology.orchestra.run.smithi007.stdout:Package: linux-image-generic 2023-12-16T07:18:32.846 INFO:teuthology.orchestra.run.smithi007.stdout:Status: install ok installed 2023-12-16T07:18:32.846 INFO:teuthology.orchestra.run.smithi007.stdout:Priority: optional 2023-12-16T07:18:32.846 INFO:teuthology.orchestra.run.smithi007.stdout:Section: kernel 2023-12-16T07:18:32.847 INFO:teuthology.orchestra.run.smithi007.stdout:Installed-Size: 20 2023-12-16T07:18:32.847 INFO:teuthology.orchestra.run.smithi007.stdout:Maintainer: Ubuntu Kernel Team 2023-12-16T07:18:32.847 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: amd64 2023-12-16T07:18:32.847 INFO:teuthology.orchestra.run.smithi007.stdout:Source: linux-meta 2023-12-16T07:18:32.847 INFO:teuthology.orchestra.run.smithi007.stdout:Version: 5.4.0.169.167 2023-12-16T07:18:32.847 INFO:teuthology.orchestra.run.smithi007.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-12-16T07:18:32.847 INFO:teuthology.orchestra.run.smithi007.stdout:Depends: linux-image-5.4.0-169-generic, linux-modules-extra-5.4.0-169-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-16T07:18:32.847 INFO:teuthology.orchestra.run.smithi007.stdout:Recommends: thermald 2023-12-16T07:18:32.847 INFO:teuthology.orchestra.run.smithi007.stdout:Description: Generic Linux kernel image 2023-12-16T07:18:32.847 INFO:teuthology.orchestra.run.smithi007.stdout: This package will always depend on the latest generic kernel image 2023-12-16T07:18:32.847 INFO:teuthology.orchestra.run.smithi007.stdout: available. 2023-12-16T07:18:32.848 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-169-generic 2023-12-16T07:18:32.848 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-16T07:18:32.848 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-16T07:18:32.848 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-16T07:18:32.848 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2023-12-16T07:18:32.848 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get clean 2023-12-16T07:18:32.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo apt-get update 2023-12-16T07:18:33.030 INFO:teuthology.orchestra.run.smithi169.stderr:Found linux image: /boot/vmlinuz-5.4.0-169-generic 2023-12-16T07:18:33.042 INFO:teuthology.orchestra.run.smithi169.stderr:Found initrd image: /boot/initrd.img-5.4.0-169-generic 2023-12-16T07:18:33.065 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-12-16T07:18:33.073 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-12-16T07:18:33.091 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-12-16T07:18:33.200 INFO:teuthology.orchestra.run.smithi007.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-12-16T07:18:33.374 INFO:teuthology.orchestra.run.smithi169.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-16T07:18:33.385 INFO:teuthology.orchestra.run.smithi169.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-16T07:18:34.321 INFO:teuthology.orchestra.run.smithi169.stderr:done 2023-12-16T07:18:34.323 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2023-12-16T07:18:34.883 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2023-12-16T07:18:34.911 DEBUG:teuthology.orchestra.run.smithi007:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-16T07:18:34.981 INFO:teuthology.orchestra.run.smithi007.stdout:Reading package lists... 2023-12-16T07:18:35.198 INFO:teuthology.orchestra.run.smithi007.stdout:Building dependency tree... 2023-12-16T07:18:35.199 INFO:teuthology.orchestra.run.smithi007.stdout:Reading state information... 2023-12-16T07:18:35.444 INFO:teuthology.orchestra.run.smithi007.stdout:linux-image-generic is already the newest version (5.4.0.169.167). 2023-12-16T07:18:35.444 INFO:teuthology.orchestra.run.smithi007.stdout:0 upgraded, 0 newly installed, 0 to remove and 280 not upgraded. 2023-12-16T07:18:35.446 DEBUG:teuthology.orchestra.run.smithi007:> dpkg -s linux-image-generic 2023-12-16T07:18:35.467 INFO:teuthology.orchestra.run.smithi007.stdout:Package: linux-image-generic 2023-12-16T07:18:35.467 INFO:teuthology.orchestra.run.smithi007.stdout:Status: install ok installed 2023-12-16T07:18:35.467 INFO:teuthology.orchestra.run.smithi007.stdout:Priority: optional 2023-12-16T07:18:35.467 INFO:teuthology.orchestra.run.smithi007.stdout:Section: kernel 2023-12-16T07:18:35.468 INFO:teuthology.orchestra.run.smithi007.stdout:Installed-Size: 20 2023-12-16T07:18:35.468 INFO:teuthology.orchestra.run.smithi007.stdout:Maintainer: Ubuntu Kernel Team 2023-12-16T07:18:35.468 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: amd64 2023-12-16T07:18:35.468 INFO:teuthology.orchestra.run.smithi007.stdout:Source: linux-meta 2023-12-16T07:18:35.468 INFO:teuthology.orchestra.run.smithi007.stdout:Version: 5.4.0.169.167 2023-12-16T07:18:35.468 INFO:teuthology.orchestra.run.smithi007.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.15) 2023-12-16T07:18:35.468 INFO:teuthology.orchestra.run.smithi007.stdout:Depends: linux-image-5.4.0-169-generic, linux-modules-extra-5.4.0-169-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-16T07:18:35.468 INFO:teuthology.orchestra.run.smithi007.stdout:Recommends: thermald 2023-12-16T07:18:35.468 INFO:teuthology.orchestra.run.smithi007.stdout:Description: Generic Linux kernel image 2023-12-16T07:18:35.468 INFO:teuthology.orchestra.run.smithi007.stdout: This package will always depend on the latest generic kernel image 2023-12-16T07:18:35.468 INFO:teuthology.orchestra.run.smithi007.stdout: available. 2023-12-16T07:18:35.469 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2023-12-16T07:18:35.517 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.zAiiOfLpDn 2023-12-16T07:18:35.518 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub/grub.cfg /tmp/tmp.zAiiOfLpDn 2023-12-16T07:18:35.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.zAiiOfLpDn 2023-12-16T07:18:35.588 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.zAiiOfLpDn is 9KB 2023-12-16T07:18:35.601 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.zAiiOfLpDn 2023-12-16T07:18:35.607 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-16T07:18:35.666 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-12-16T07:18:35.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-16T07:18:35.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-16T07:18:35.695 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-169-generic 2023-12-16T07:18:35.695 DEBUG:teuthology.orchestra.run.smithi007:> sudo update-grub 2023-12-16T07:18:37.114 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub' 2023-12-16T07:18:37.125 INFO:teuthology.orchestra.run.smithi007.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-16T07:18:37.139 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2023-12-16T07:18:37.421 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.4.0-169-generic 2023-12-16T07:18:37.433 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.4.0-169-generic 2023-12-16T07:18:37.823 INFO:teuthology.orchestra.run.smithi007.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-12-16T07:18:37.835 INFO:teuthology.orchestra.run.smithi007.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-12-16T07:18:38.683 INFO:teuthology.orchestra.run.smithi007.stderr:done 2023-12-16T07:18:38.685 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2023-12-16T07:19:04.349 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-16T07:19:04.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T07:19:04.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T07:19:08.693 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-16T07:19:08.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-12-16T07:19:08.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T07:19:39.026 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2023-12-16T07:19:48.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T07:19:48.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T07:20:08.714 DEBUG:teuthology.orchestra.remote:timed out 2023-12-16T07:20:17.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-12-16T07:20:17.722 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T07:20:20.755 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2023-12-16T07:20:32.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-12-16T07:20:32.762 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T07:20:32.968 DEBUG:teuthology.orchestra.run.smithi007:> true 2023-12-16T07:20:33.578 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-12-16T07:20:33.578 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-169-generic"... 2023-12-16T07:20:33.578 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-12-16T07:20:33.585 INFO:teuthology.orchestra.run.smithi007.stdout:5.4.0-169-generic 2023-12-16T07:20:33.585 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-169-generic vs 5.4.0-169-generic 2023-12-16T07:20:33.585 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-16T07:20:33.585 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-16T07:20:34.586 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-16T07:20:34.587 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-16T07:20:34.683 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2023-12-16T07:20:34.706 DEBUG:teuthology.parallel:result is None 2023-12-16T07:20:48.041 DEBUG:teuthology.orchestra.remote:timed out 2023-12-16T07:21:00.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T07:21:00.043 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T07:21:00.244 DEBUG:teuthology.orchestra.run.smithi169:> true 2023-12-16T07:21:00.840 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T07:21:00.840 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-169-generic"... 2023-12-16T07:21:00.840 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2023-12-16T07:21:00.847 INFO:teuthology.orchestra.run.smithi169.stdout:5.4.0-169-generic 2023-12-16T07:21:00.847 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-169-generic vs 5.4.0-169-generic 2023-12-16T07:21:00.847 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-16T07:21:00.847 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-16T07:21:01.848 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-16T07:21:01.848 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-16T07:21:01.952 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2023-12-16T07:21:01.983 DEBUG:teuthology.parallel:result is None 2023-12-16T07:21:01.983 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-16T07:21:01.991 INFO:teuthology.task.internal:Creating test directory... 2023-12-16T07:21:01.991 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-16T07:21:01.994 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-16T07:21:02.001 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-16T07:21:02.008 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-16T07:21:02.016 INFO:teuthology.task.internal:Creating archive directory... 2023-12-16T07:21:02.016 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-16T07:21:02.019 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-16T07:21:02.044 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-16T07:21:02.052 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-16T07:21:02.052 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-16T07:21:02.081 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-16T07:21:02.098 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T07:21:02.101 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T07:21:02.108 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T07:21:02.112 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-16T07:21:02.114 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-16T07:21:02.122 INFO:teuthology.task.internal:Configuring sudo... 2023-12-16T07:21:02.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-16T07:21:02.124 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-16T07:21:02.141 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-16T07:21:02.150 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-16T07:21:02.151 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-16T07:21:02.153 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-16T07:21:02.159 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T07:21:02.206 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T07:21:02.214 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-12-16T07:21:02.214 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-16T07:21:02.238 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-16T07:21:02.245 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-16T07:21:02.252 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T07:21:02.252 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-16T07:21:02.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2023-12-16T07:21:02.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2023-12-16T07:21:02.383 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-16T07:21:02.392 INFO:teuthology.task.internal:Starting timer... 2023-12-16T07:21:02.392 INFO:teuthology.run_tasks:Running task pcp... 2023-12-16T07:21:02.402 INFO:teuthology.run_tasks:Running task selinux... 2023-12-16T07:21:02.410 DEBUG:teuthology.task.selinux:Excluding smithi007: OS 'ubuntu' does not support SELinux 2023-12-16T07:21:02.410 DEBUG:teuthology.task.selinux:Excluding smithi169: OS 'ubuntu' does not support SELinux 2023-12-16T07:21:02.411 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-16T07:21:02.411 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-12-16T07:21:02.411 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-16T07:21:02.411 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-16T07:21:02.419 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-16T07:21:02.419 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-16T07:21:02.436 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-16T07:21:02.438 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi169.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-16T07:27:10.382 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi169.front.sepia.ceph.com')] 2023-12-16T07:27:10.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-12-16T07:27:10.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T07:27:10.439 DEBUG:teuthology.orchestra.run.smithi007:> true 2023-12-16T07:27:10.559 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-12-16T07:27:10.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T07:27:10.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-16T07:27:10.609 DEBUG:teuthology.orchestra.run.smithi169:> true 2023-12-16T07:27:10.731 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2023-12-16T07:27:10.731 INFO:teuthology.run_tasks:Running task clock... 2023-12-16T07:27:10.743 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-16T07:27:10.743 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-16T07:27:10.744 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T07:27:10.746 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-16T07:27:10.746 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T07:27:10.777 INFO:teuthology.orchestra.run.smithi007.stdout:16 Dec 07:27:10 ntpd[23546]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-12-16T07:27:10.778 INFO:teuthology.orchestra.run.smithi007.stdout:16 Dec 07:27:10 ntpd[23546]: Command line: ntpd -gq 2023-12-16T07:27:10.779 INFO:teuthology.orchestra.run.smithi007.stdout:16 Dec 07:27:10 ntpd[23546]: proto: precision = 0.103 usec (-23) 2023-12-16T07:27:10.779 INFO:teuthology.orchestra.run.smithi007.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-16T07:27:10.780 INFO:teuthology.orchestra.run.smithi007.stdout:16 Dec 07:27:10 ntpd[23546]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-16T07:27:10.780 INFO:teuthology.orchestra.run.smithi007.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-16T07:27:10.781 INFO:teuthology.orchestra.run.smithi007.stdout:16 Dec 07:27:10 ntpd[23546]: restrict ::: KOD does nothing without LIMITED. 2023-12-16T07:27:10.781 INFO:teuthology.orchestra.run.smithi007.stdout:16 Dec 07:27:10 ntpd[23546]: Listen and drop on 0 v6wildcard [::]:123 2023-12-16T07:27:10.781 INFO:teuthology.orchestra.run.smithi007.stdout:16 Dec 07:27:10 ntpd[23546]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-16T07:27:10.781 INFO:teuthology.orchestra.run.smithi007.stdout:16 Dec 07:27:10 ntpd[23546]: Listen normally on 2 lo 127.0.0.1:123 2023-12-16T07:27:10.781 INFO:teuthology.orchestra.run.smithi007.stdout:16 Dec 07:27:10 ntpd[23546]: Listen normally on 3 ens1f0 172.21.15.7:123 2023-12-16T07:27:10.782 INFO:teuthology.orchestra.run.smithi007.stdout:16 Dec 07:27:10 ntpd[23546]: Listen normally on 4 lo [::1]:123 2023-12-16T07:27:10.782 INFO:teuthology.orchestra.run.smithi007.stdout:16 Dec 07:27:10 ntpd[23546]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15ea%4]:123 2023-12-16T07:27:10.782 INFO:teuthology.orchestra.run.smithi007.stdout:16 Dec 07:27:10 ntpd[23546]: Listening on routing socket on fd #22 for interface updates 2023-12-16T07:27:10.782 INFO:teuthology.orchestra.run.smithi169.stdout:16 Dec 07:27:10 ntpd[23610]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-12-16T07:27:10.782 INFO:teuthology.orchestra.run.smithi169.stdout:16 Dec 07:27:10 ntpd[23610]: Command line: ntpd -gq 2023-12-16T07:27:10.784 INFO:teuthology.orchestra.run.smithi169.stdout:16 Dec 07:27:10 ntpd[23610]: proto: precision = 0.117 usec (-23) 2023-12-16T07:27:10.784 INFO:teuthology.orchestra.run.smithi169.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-16T07:27:10.785 INFO:teuthology.orchestra.run.smithi169.stdout:16 Dec 07:27:10 ntpd[23610]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-16T07:27:10.785 INFO:teuthology.orchestra.run.smithi169.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-16T07:27:10.786 INFO:teuthology.orchestra.run.smithi169.stdout:16 Dec 07:27:10 ntpd[23610]: restrict ::: KOD does nothing without LIMITED. 2023-12-16T07:27:10.786 INFO:teuthology.orchestra.run.smithi169.stdout:16 Dec 07:27:10 ntpd[23610]: Listen and drop on 0 v6wildcard [::]:123 2023-12-16T07:27:10.786 INFO:teuthology.orchestra.run.smithi169.stdout:16 Dec 07:27:10 ntpd[23610]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-16T07:27:10.786 INFO:teuthology.orchestra.run.smithi169.stdout:16 Dec 07:27:10 ntpd[23610]: Listen normally on 2 lo 127.0.0.1:123 2023-12-16T07:27:10.786 INFO:teuthology.orchestra.run.smithi169.stdout:16 Dec 07:27:10 ntpd[23610]: Listen normally on 3 enp3s0f1 172.21.15.169:123 2023-12-16T07:27:10.786 INFO:teuthology.orchestra.run.smithi169.stdout:16 Dec 07:27:10 ntpd[23610]: Listen normally on 4 lo [::1]:123 2023-12-16T07:27:10.786 INFO:teuthology.orchestra.run.smithi169.stdout:16 Dec 07:27:10 ntpd[23610]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f4d%5]:123 2023-12-16T07:27:10.786 INFO:teuthology.orchestra.run.smithi169.stdout:16 Dec 07:27:10 ntpd[23610]: Listening on routing socket on fd #22 for interface updates 2023-12-16T07:27:11.782 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:11 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:11.782 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:11 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:11.782 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:11 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:11.782 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:11 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:11.782 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:11 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:11.783 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:11 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:11.783 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:11 ntpd[23546]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:11.783 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:11 ntpd[23546]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:11.783 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:11 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:11.783 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:11 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:11.783 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:11 ntpd[23546]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:11.783 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:11 ntpd[23546]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:12.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:12 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:12.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:12 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:12.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:12 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:12.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:12 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:12.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:12 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:12.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:12 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:12.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:12 ntpd[23610]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:12.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:12 ntpd[23610]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:12.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:12 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:12.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:12 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:12.785 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:12 ntpd[23610]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:12.785 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:12 ntpd[23610]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:13.779 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:13 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:13.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:13 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:13.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:13 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:13.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:13 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:13.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:13 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:13.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:13 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:13.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:13 ntpd[23546]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:13.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:13 ntpd[23546]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:13.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:13 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:13.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:13 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:13.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:13 ntpd[23546]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:13.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:13 ntpd[23546]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:14 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:14.778 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:14 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:14.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:14 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:14.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:14 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:14.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:14 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:14.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:14 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:14.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:14 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:14.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:14 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:14.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:14 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:14.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:14 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:14.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:14 ntpd[23610]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:14.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:14 ntpd[23610]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:14.785 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:14 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:14.785 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:14 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:14.785 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:14 ntpd[23610]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:14.785 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:14 ntpd[23610]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:15.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:15.781 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:15 ntpd[23546]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:15.783 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:15 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:15.783 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:15 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:15.783 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:15 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:15.783 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:15 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:15.783 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:15 ntpd[23610]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:15.783 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:15 ntpd[23610]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:16.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:16 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:16.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:16 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:16.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:16 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:16.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:16 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:16.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:16 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:16.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:16 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:16.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:16 ntpd[23610]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:16.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:16 ntpd[23610]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:16.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:16 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:16.785 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:16 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:16.785 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:16 ntpd[23610]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:16.785 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:16 ntpd[23610]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:17 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:17 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:17 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:17 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:17 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:17 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:17 ntpd[23546]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:17 ntpd[23546]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:17 ntpd[23546]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:17.779 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:17 ntpd[23546]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:17.780 INFO:teuthology.orchestra.run.smithi007.stdout:16 Dec 07:27:17 ntpd[23546]: ntpd: time slew +0.000736 s 2023-12-16T07:27:17.780 INFO:teuthology.orchestra.run.smithi007.stdout:ntpd: time slew +0.000736s 2023-12-16T07:27:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:17 ntpd[23546]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-16T07:27:17.780 INFO:teuthology.orchestra.run.smithi007.stderr:16 Dec 07:27:17 ntpd[23546]: can't open /var/log/ntpstats/loopstats.20231216: Permission denied 2023-12-16T07:27:17.783 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:17 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:17.783 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:17 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:17.783 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:17 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:17.783 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:17 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:17.783 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:17 ntpd[23610]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:17.783 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:17 ntpd[23610]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: remote refid st t when poll reach delay offset jitter 2023-12-16T07:27:17.846 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================== 2023-12-16T07:27:17.846 INFO:teuthology.orchestra.run.smithi007.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-16T07:27:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-16T07:27:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-16T07:27:17.847 INFO:teuthology.orchestra.run.smithi007.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-16T07:27:18.783 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:18 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:18.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:18 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:18.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:18 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:18.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:18 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:18.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:18 ntpd[23610]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-16T07:27:18.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:18 ntpd[23610]: can't open /var/log/ntpstats/rawstats.20231216: Permission denied 2023-12-16T07:27:18.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:18 ntpd[23610]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-16T07:27:18.784 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:18 ntpd[23610]: can't open /var/log/ntpstats/peerstats.20231216: Permission denied 2023-12-16T07:27:18.784 INFO:teuthology.orchestra.run.smithi169.stdout:16 Dec 07:27:18 ntpd[23610]: ntpd: time slew +0.000704 s 2023-12-16T07:27:18.784 INFO:teuthology.orchestra.run.smithi169.stdout:ntpd: time slew +0.000704s 2023-12-16T07:27:18.785 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:18 ntpd[23610]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-16T07:27:18.785 INFO:teuthology.orchestra.run.smithi169.stderr:16 Dec 07:27:18 ntpd[23610]: can't open /var/log/ntpstats/loopstats.20231216: Permission denied 2023-12-16T07:27:18.860 INFO:teuthology.orchestra.run.smithi169.stdout: remote refid st t when poll reach delay offset jitter 2023-12-16T07:27:18.860 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================== 2023-12-16T07:27:18.860 INFO:teuthology.orchestra.run.smithi169.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-16T07:27:18.860 INFO:teuthology.orchestra.run.smithi169.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-16T07:27:18.860 INFO:teuthology.orchestra.run.smithi169.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-16T07:27:18.860 INFO:teuthology.orchestra.run.smithi169.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-12-16T07:27:18.861 INFO:teuthology.run_tasks:Running task cephadm... 2023-12-16T07:27:18.939 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '6d933094a156ec7587742e5aa25fba5ba27ec678'} 2023-12-16T07:27:18.940 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2023-12-16T07:27:18.940 INFO:tasks.cephadm:Cluster fsid is 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:27:18.940 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-12-16T07:27:18.940 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.7', 'mon.c': '[v2:172.21.15.7:3301,v1:172.21.15.7:6790]', 'mon.b': '172.21.15.169'} 2023-12-16T07:27:18.941 INFO:tasks.cephadm:First mon is mon.a on smithi007 2023-12-16T07:27:18.941 INFO:tasks.cephadm:First mgr is y 2023-12-16T07:27:18.941 INFO:tasks.cephadm:Normalizing hostnames... 2023-12-16T07:27:18.941 DEBUG:teuthology.orchestra.run.smithi007:> sudo hostname $(hostname -s) 2023-12-16T07:27:18.958 DEBUG:teuthology.orchestra.run.smithi169:> sudo hostname $(hostname -s) 2023-12-16T07:27:18.975 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2023-12-16T07:27:18.975 DEBUG:teuthology.orchestra.run.smithi007:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-16T07:27:19.284 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-rw-r-- 1 ubuntu ubuntu 320521 Dec 16 07:27 /home/ubuntu/cephtest/cephadm 2023-12-16T07:27:19.285 DEBUG:teuthology.orchestra.run.smithi169:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-16T07:27:19.462 INFO:teuthology.orchestra.run.smithi169.stdout:-rw-rw-r-- 1 ubuntu ubuntu 320521 Dec 16 07:27 /home/ubuntu/cephtest/cephadm 2023-12-16T07:27:19.462 DEBUG:teuthology.orchestra.run.smithi007:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-16T07:27:19.471 DEBUG:teuthology.orchestra.run.smithi169:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-16T07:27:19.491 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2023-12-16T07:27:19.491 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2023-12-16T07:27:19.493 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2023-12-16T07:27:19.783 INFO:teuthology.orchestra.run.smithi007.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2023-12-16T07:27:19.784 INFO:teuthology.orchestra.run.smithi169.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2023-12-16T07:27:55.367 INFO:teuthology.orchestra.run.smithi169.stdout:{ 2023-12-16T07:27:55.368 INFO:teuthology.orchestra.run.smithi169.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2023-12-16T07:27:55.368 INFO:teuthology.orchestra.run.smithi169.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2023-12-16T07:27:55.368 INFO:teuthology.orchestra.run.smithi169.stdout: "repo_digests": [ 2023-12-16T07:27:55.368 INFO:teuthology.orchestra.run.smithi169.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a" 2023-12-16T07:27:55.368 INFO:teuthology.orchestra.run.smithi169.stdout: ] 2023-12-16T07:27:55.368 INFO:teuthology.orchestra.run.smithi169.stdout:} 2023-12-16T07:27:56.856 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:27:56.856 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2023-12-16T07:27:56.856 INFO:teuthology.orchestra.run.smithi007.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2023-12-16T07:27:56.856 INFO:teuthology.orchestra.run.smithi007.stdout: "repo_digests": [ 2023-12-16T07:27:56.856 INFO:teuthology.orchestra.run.smithi007.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a" 2023-12-16T07:27:56.856 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2023-12-16T07:27:56.856 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:27:56.881 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph 2023-12-16T07:27:56.895 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph 2023-12-16T07:27:56.912 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /etc/ceph 2023-12-16T07:27:56.926 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /etc/ceph 2023-12-16T07:27:56.940 INFO:tasks.cephadm:Writing seed config... 2023-12-16T07:27:56.941 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2023-12-16T07:27:56.941 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-12-16T07:27:56.941 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-12-16T07:27:56.941 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2023-12-16T07:27:56.941 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-12-16T07:27:56.941 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-12-16T07:27:56.941 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-12-16T07:27:56.941 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-12-16T07:27:56.942 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-12-16T07:27:56.942 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-12-16T07:27:56.942 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-12-16T07:27:56.947 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8b71d3dc-9be4-11ee-95a5-87774f69a715 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2023-12-16T07:27:56.948 DEBUG:teuthology.orchestra.run.smithi007:mon.a> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.a.service 2023-12-16T07:27:56.992 DEBUG:teuthology.orchestra.run.smithi007:mgr.y> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.y.service 2023-12-16T07:27:56.994 INFO:tasks.cephadm:Bootstrapping... 2023-12-16T07:27:56.994 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.7 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-12-16T07:27:57.158 INFO:teuthology.orchestra.run.smithi007.stderr:-------------------------------------------------------------------------------- 2023-12-16T07:27:57.158 INFO:teuthology.orchestra.run.smithi007.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '8b71d3dc-9be4-11ee-95a5-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.7', '--skip-admin-label'] 2023-12-16T07:27:57.158 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying podman|docker is present... 2023-12-16T07:27:57.158 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying lvm2 is present... 2023-12-16T07:27:57.158 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying time synchronization is in place... 2023-12-16T07:27:57.163 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-16T07:27:57.171 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2023-12-16T07:27:57.178 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2023-12-16T07:27:57.184 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2023-12-16T07:27:57.189 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: masked 2023-12-16T07:27:57.197 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2023-12-16T07:27:57.204 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2023-12-16T07:27:57.212 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2023-12-16T07:27:57.219 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: enabled 2023-12-16T07:27:57.225 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: active 2023-12-16T07:27:57.227 INFO:teuthology.orchestra.run.smithi007.stderr:Unit ntp.service is enabled and running 2023-12-16T07:27:57.227 INFO:teuthology.orchestra.run.smithi007.stderr:Repeating the final host check... 2023-12-16T07:27:57.227 INFO:teuthology.orchestra.run.smithi007.stderr:docker (/usr/bin/docker) is present 2023-12-16T07:27:57.227 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl is present 2023-12-16T07:27:57.227 INFO:teuthology.orchestra.run.smithi007.stderr:lvcreate is present 2023-12-16T07:27:57.232 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-16T07:27:57.239 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2023-12-16T07:27:57.245 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to get unit file state for chronyd.service: No such file or directory 2023-12-16T07:27:57.251 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2023-12-16T07:27:57.258 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: masked 2023-12-16T07:27:57.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:27:57.264 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2023-12-16T07:27:57.271 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to get unit file state for ntpd.service: No such file or directory 2023-12-16T07:27:57.277 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2023-12-16T07:27:57.279 INFO:journalctl@ceph.mgr.y.smithi007.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:27:57.286 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: enabled 2023-12-16T07:27:57.293 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: active 2023-12-16T07:27:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:Unit ntp.service is enabled and running 2023-12-16T07:27:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:Host looks OK 2023-12-16T07:27:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:Cluster fsid: 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:27:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:Acquiring lock 140242090064192 on /run/cephadm/8b71d3dc-9be4-11ee-95a5-87774f69a715.lock 2023-12-16T07:27:57.294 INFO:teuthology.orchestra.run.smithi007.stderr:Lock 140242090064192 acquired on /run/cephadm/8b71d3dc-9be4-11ee-95a5-87774f69a715.lock 2023-12-16T07:27:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying IP 172.21.15.7 port 3300 ... 2023-12-16T07:27:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying IP 172.21.15.7 port 6789 ... 2023-12-16T07:27:57.295 INFO:teuthology.orchestra.run.smithi007.stderr:Base mon IP is 172.21.15.7, final addrv is [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2023-12-16T07:27:57.298 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/sbin/ip: default via 172.21.15.254 dev ens1f0 2023-12-16T07:27:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/sbin/ip: 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 2023-12-16T07:27:57.299 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.7 2023-12-16T07:27:57.303 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-12-16T07:27:57.304 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-12-16T07:27:57.304 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1797sec hoplimit 64 pref medium 2023-12-16T07:27:57.307 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-12-16T07:27:57.308 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/sbin/ip: inet6 ::1/128 scope host 2023-12-16T07:27:57.308 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2023-12-16T07:27:57.308 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2023-12-16T07:27:57.308 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/sbin/ip: inet6 fe80::ec4:7aff:febd:15ea/64 scope link 2023-12-16T07:27:57.308 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/sbin/ip: valid_lft forever preferred_lft forever 2023-12-16T07:27:57.309 INFO:teuthology.orchestra.run.smithi007.stderr:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2023-12-16T07:27:57.309 INFO:teuthology.orchestra.run.smithi007.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-12-16T07:27:57.310 INFO:teuthology.orchestra.run.smithi007.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2023-12-16T07:27:57.716 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/docker: v17.2.0: Pulling from ceph/ceph 2023-12-16T07:27:57.847 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/docker: Digest: sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2023-12-16T07:27:57.847 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/docker: Status: Image is up to date for quay.io/ceph/ceph:v17.2.0 2023-12-16T07:27:57.905 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/docker: quay.io/ceph/ceph:v17.2.0 2023-12-16T07:27:58.913 INFO:teuthology.orchestra.run.smithi007.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2023-12-16T07:27:59.501 INFO:teuthology.orchestra.run.smithi007.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2023-12-16T07:27:59.501 INFO:teuthology.orchestra.run.smithi007.stderr:Extracting ceph user uid/gid from container image... 2023-12-16T07:28:00.328 INFO:teuthology.orchestra.run.smithi007.stderr:stat: 167 167 2023-12-16T07:28:00.922 INFO:teuthology.orchestra.run.smithi007.stderr:Creating initial keys... 2023-12-16T07:28:01.723 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-authtool: AQCBUX1lP9gYKxAAXmB2utPOHFQa5fzGDBDbag== 2023-12-16T07:28:03.186 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-authtool: AQCDUX1lUVoRCxAA351SwvFBsf1sAgd9uLTNBA== 2023-12-16T07:28:04.595 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-authtool: AQCEUX1ls0Z4IxAAxgE9AoXBDQwn3vHun2AmWA== 2023-12-16T07:28:05.229 INFO:teuthology.orchestra.run.smithi007.stderr:Creating initial monmap... 2023-12-16T07:28:06.054 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-16T07:28:06.054 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2023-12-16T07:28:06.055 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:28:06.055 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-16T07:28:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:monmaptool for a [v2:172.21.15.7:3300,v1:172.21.15.7:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-16T07:28:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:setting min_mon_release = octopus 2023-12-16T07:28:06.658 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: set fsid to 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:28:06.659 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-16T07:28:06.659 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-12-16T07:28:06.659 INFO:teuthology.orchestra.run.smithi007.stderr:Creating mon... 2023-12-16T07:28:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.455+0000 7f177b8ae880 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-16T07:28:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.455+0000 7f177b8ae880 1 imported monmap: 2023-12-16T07:28:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: epoch 0 2023-12-16T07:28:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:28:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: last_changed 2023-12-16T07:28:06.053524+0000 2023-12-16T07:28:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: created 2023-12-16T07:28:06.053524+0000 2023-12-16T07:28:07.463 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: min_mon_release 15 (octopus) 2023-12-16T07:28:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-12-16T07:28:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-12-16T07:28:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.464 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.455+0000 7f177b8ae880 0 /usr/bin/ceph-mon: set fsid to 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:28:07.465 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: RocksDB version: 6.15.5 2023-12-16T07:28:07.465 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.466 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-16T07:28:07.466 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Compile date Apr 18 2022 2023-12-16T07:28:07.466 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: DB SUMMARY 2023-12-16T07:28:07.466 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.466 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: DB Session ID: OQ5L9U78KL2A6XS5667H 2023-12-16T07:28:07.467 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.467 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2023-12-16T07:28:07.470 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2023-12-16T07:28:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.error_if_exists: 0 2023-12-16T07:28:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.create_if_missing: 1 2023-12-16T07:28:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.paranoid_checks: 1 2023-12-16T07:28:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-12-16T07:28:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.env: 0x564172304860 2023-12-16T07:28:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.fs: Posix File System 2023-12-16T07:28:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.info_log: 0x5641744e3320 2023-12-16T07:28:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-16T07:28:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.statistics: (nil) 2023-12-16T07:28:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.use_fsync: 0 2023-12-16T07:28:07.471 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.max_log_file_size: 0 2023-12-16T07:28:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-16T07:28:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-16T07:28:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-16T07:28:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-16T07:28:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.allow_fallocate: 1 2023-12-16T07:28:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-16T07:28:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-16T07:28:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.use_direct_reads: 0 2023-12-16T07:28:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-16T07:28:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.create_missing_column_families: 0 2023-12-16T07:28:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.db_log_dir: 2023-12-16T07:28:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2023-12-16T07:28:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-16T07:28:07.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-16T07:28:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-16T07:28:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-16T07:28:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-16T07:28:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-16T07:28:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.advise_random_on_open: 1 2023-12-16T07:28:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-16T07:28:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.write_buffer_manager: 0x564174783950 2023-12-16T07:28:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-16T07:28:07.473 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-16T07:28:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-16T07:28:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-16T07:28:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.rate_limiter: (nil) 2023-12-16T07:28:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-16T07:28:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-16T07:28:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-16T07:28:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-16T07:28:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.unordered_write: 0 2023-12-16T07:28:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-16T07:28:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-16T07:28:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-16T07:28:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-16T07:28:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.row_cache: None 2023-12-16T07:28:07.480 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.wal_filter: None 2023-12-16T07:28:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-16T07:28:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-16T07:28:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.preserve_deletes: 0 2023-12-16T07:28:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.two_write_queues: 0 2023-12-16T07:28:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.manual_wal_flush: 0 2023-12-16T07:28:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.atomic_flush: 0 2023-12-16T07:28:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-16T07:28:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-16T07:28:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-16T07:28:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.log_readahead_size: 0 2023-12-16T07:28:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-12-16T07:28:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.best_efforts_recovery: 0 2023-12-16T07:28:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-12-16T07:28:07.481 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-12-16T07:28:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.allow_data_in_errors: 0 2023-12-16T07:28:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.db_host_id: __hostname__ 2023-12-16T07:28:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.max_background_jobs: 2 2023-12-16T07:28:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.max_background_compactions: -1 2023-12-16T07:28:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.max_subcompactions: 1 2023-12-16T07:28:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-16T07:28:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-16T07:28:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-16T07:28:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.max_total_wal_size: 0 2023-12-16T07:28:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-16T07:28:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-16T07:28:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-16T07:28:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-16T07:28:07.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.max_open_files: -1 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.bytes_per_sync: 0 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Options.max_background_flushes: -1 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Compression algorithms supported: 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: kZSTD supported: 0 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: kXpressCompression supported: 0 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: kLZ4Compression supported: 1 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: kBZip2Compression supported: 0 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: kZlibCompression supported: 1 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: kSnappyCompression supported: 1 2023-12-16T07:28:07.483 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.459+0000 7f177b8ae880 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-16T07:28:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.463+0000 7f177b8ae880 4 rocksdb: [db/db_impl/db_impl_open.cc:281] Creating manifest 1 2023-12-16T07:28:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2023-12-16T07:28:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2023-12-16T07:28:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-16T07:28:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.merge_operator: 2023-12-16T07:28:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compaction_filter: None 2023-12-16T07:28:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compaction_filter_factory: None 2023-12-16T07:28:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.sst_partitioner_factory: None 2023-12-16T07:28:07.484 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-16T07:28:07.501 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-16T07:28:07.501 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5641744acd10) 2023-12-16T07:28:07.501 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-12-16T07:28:07.501 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-16T07:28:07.501 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-16T07:28:07.501 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-12-16T07:28:07.501 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: index_type: 0 2023-12-16T07:28:07.501 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-12-16T07:28:07.501 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-12-16T07:28:07.501 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-12-16T07:28:07.501 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-12-16T07:28:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: checksum: 1 2023-12-16T07:28:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-12-16T07:28:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: block_cache: 0x564174514170 2023-12-16T07:28:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-12-16T07:28:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-12-16T07:28:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-12-16T07:28:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-12-16T07:28:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-12-16T07:28:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-12-16T07:28:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-12-16T07:28:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-12-16T07:28:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-12-16T07:28:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-12-16T07:28:07.502 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: format_version: 4 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: block_align: 0 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-16T07:28:07.503 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compression: NoCompression 2023-12-16T07:28:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-16T07:28:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-16T07:28:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-16T07:28:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.num_levels: 7 2023-12-16T07:28:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-16T07:28:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-16T07:28:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-16T07:28:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-16T07:28:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-16T07:28:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-16T07:28:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-16T07:28:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-16T07:28:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-12-16T07:28:07.504 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-16T07:28:07.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-16T07:28:07.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compression_opts.level: 32767 2023-12-16T07:28:07.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-16T07:28:07.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-16T07:28:07.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-16T07:28:07.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-12-16T07:28:07.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compression_opts.enabled: false 2023-12-16T07:28:07.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-16T07:28:07.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-16T07:28:07.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-16T07:28:07.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-16T07:28:07.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-16T07:28:07.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-16T07:28:07.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-16T07:28:07.506 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-16T07:28:07.506 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-16T07:28:07.506 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-16T07:28:07.506 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-16T07:28:07.506 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-16T07:28:07.506 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-16T07:28:07.506 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-16T07:28:07.506 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-16T07:28:07.506 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-16T07:28:07.506 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-16T07:28:07.506 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.arena_block_size: 4194304 2023-12-16T07:28:07.506 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-16T07:28:07.506 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-16T07:28:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-16T07:28:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-16T07:28:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-16T07:28:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-16T07:28:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-16T07:28:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-16T07:28:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-16T07:28:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-16T07:28:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-16T07:28:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-16T07:28:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-16T07:28:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-16T07:28:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.table_properties_collectors: 2023-12-16T07:28:07.507 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.inplace_update_support: 0 2023-12-16T07:28:07.508 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-16T07:28:07.508 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-16T07:28:07.508 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-16T07:28:07.508 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-16T07:28:07.508 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.bloom_locality: 0 2023-12-16T07:28:07.508 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.max_successive_merges: 0 2023-12-16T07:28:07.508 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-16T07:28:07.508 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-16T07:28:07.508 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.force_consistency_checks: 1 2023-12-16T07:28:07.508 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-16T07:28:07.508 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.ttl: 2592000 2023-12-16T07:28:07.508 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-16T07:28:07.508 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.enable_blob_files: false 2023-12-16T07:28:07.508 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.min_blob_size: 0 2023-12-16T07:28:07.509 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.blob_file_size: 268435456 2023-12-16T07:28:07.509 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.blob_compression_type: NoCompression 2023-12-16T07:28:07.509 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-12-16T07:28:07.509 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.467+0000 7f177b8ae880 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-12-16T07:28:07.509 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.471+0000 7f177b8ae880 4 rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-16T07:28:07.509 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.509 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.471+0000 7f177b8ae880 4 rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2023-12-16T07:28:07.509 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.509 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.471+0000 7f177b8ae880 4 rocksdb: [db/version_set.cc:4083] Creating manifest 3 2023-12-16T07:28:07.509 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.509 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.475+0000 7f177b8ae880 4 rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5641744fa700 2023-12-16T07:28:07.509 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.475+0000 7f177b8ae880 4 rocksdb: DB pointer 0x56417456e000 2023-12-16T07:28:07.509 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.475+0000 7f176d498700 4 rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2023-12-16T07:28:07.509 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.475+0000 7f176d498700 4 rocksdb: [db/db_impl/db_impl.cc:903] 2023-12-16T07:28:07.510 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-12-16T07:28:07.510 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:28:07.510 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-16T07:28:07.510 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-16T07:28:07.510 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T07:28:07.510 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-16T07:28:07.510 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-16T07:28:07.510 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T07:28:07.510 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.510 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-16T07:28:07.510 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:28:07.510 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:28:07.510 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-16T07:28:07.510 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.511 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-16T07:28:07.512 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.512 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-16T07:28:07.512 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:28:07.512 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:28:07.512 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-16T07:28:07.512 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-16T07:28:07.512 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.512 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-16T07:28:07.512 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:28:07.512 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:28:07.512 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:28:07.512 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T07:28:07.512 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T07:28:07.512 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T07:28:07.513 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T07:28:07.513 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-16T07:28:07.513 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:28:07.513 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:28:07.513 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T07:28:07.513 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.513 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-16T07:28:07.513 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: 2023-12-16T07:28:07.513 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.475+0000 7f177b8ae880 4 rocksdb: [db/db_impl/db_impl.cc:447] Shutdown: canceling all background work 2023-12-16T07:28:07.513 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.479+0000 7f177b8ae880 4 rocksdb: [db/db_impl/db_impl.cc:625] Shutdown complete 2023-12-16T07:28:07.513 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-mon: debug 2023-12-16T07:28:07.479+0000 7f177b8ae880 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2023-12-16T07:28:08.106 INFO:teuthology.orchestra.run.smithi007.stderr:create mon.a on 2023-12-16T07:28:08.318 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /etc/systemd/system/ceph.target. 2023-12-16T07:28:08.500 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715.target -> /etc/systemd/system/ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715.target. 2023-12-16T07:28:08.500 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715.target -> /etc/systemd/system/ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715.target. 2023-12-16T07:28:08.881 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to reset failed state of unit ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.a.service: Unit ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.a.service not loaded. 2023-12-16T07:28:08.889 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715.target.wants/ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.a.service -> /etc/systemd/system/ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@.service. 2023-12-16T07:28:09.087 INFO:teuthology.orchestra.run.smithi007.stderr:firewalld does not appear to be present 2023-12-16T07:28:09.087 INFO:teuthology.orchestra.run.smithi007.stderr:Not possible to enable service . firewalld.service is not available 2023-12-16T07:28:09.090 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mon to start... 2023-12-16T07:28:09.090 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mon... 2023-12-16T07:28:10.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:10 smithi007 bash[24530]: cluster 2023-12-16T07:28:10.352558+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2023-12-16T07:28:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: cluster: 2023-12-16T07:28:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: id: 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:28:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-12-16T07:28:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2023-12-16T07:28:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: services: 2023-12-16T07:28:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.255118s) 2023-12-16T07:28:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mgr: no daemons active 2023-12-16T07:28:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-12-16T07:28:10.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2023-12-16T07:28:10.616 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: data: 2023-12-16T07:28:10.616 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-12-16T07:28:10.616 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-12-16T07:28:10.616 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-12-16T07:28:10.616 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: pgs: 2023-12-16T07:28:10.616 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2023-12-16T07:28:11.277 INFO:teuthology.orchestra.run.smithi007.stderr:mon is available 2023-12-16T07:28:11.277 INFO:teuthology.orchestra.run.smithi007.stderr:Assimilating anything we can from ceph.conf... 2023-12-16T07:28:11.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:11 smithi007 bash[24530]: cluster 2023-12-16T07:28:10.357822+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2023-12-16T07:28:11.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:11 smithi007 bash[24530]: cluster 2023-12-16T07:28:10.358059+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]} 2023-12-16T07:28:11.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:11 smithi007 bash[24530]: cluster 2023-12-16T07:28:10.361510+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2023-12-16T07:28:11.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:11 smithi007 bash[24530]: cluster 2023-12-16T07:28:10.365512+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-16T07:28:11.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:11 smithi007 bash[24530]: cluster 2023-12-16T07:28:10.366185+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-12-16T07:28:11.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:11 smithi007 bash[24530]: audit 2023-12-16T07:28:10.612861+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.7:0/2297797018' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-12-16T07:28:12.404 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2023-12-16T07:28:12.404 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [global] 2023-12-16T07:28:12.404 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: fsid = 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:28:12.404 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2023-12-16T07:28:12.404 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-12-16T07:28:12.406 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-12-16T07:28:12.406 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-12-16T07:28:12.406 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-12-16T07:28:12.406 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2023-12-16T07:28:12.406 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [mgr] 2023-12-16T07:28:12.406 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2023-12-16T07:28:12.406 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-12-16T07:28:12.406 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2023-12-16T07:28:12.406 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [osd] 2023-12-16T07:28:12.407 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-12-16T07:28:12.407 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-12-16T07:28:12.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:12 smithi007 bash[24530]: audit 2023-12-16T07:28:12.396017+0000 mon.a (mon.0) 8 : audit [INF] from='client.? 172.21.15.7:0/2933925246' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-16T07:28:12.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:12 smithi007 bash[24530]: audit 2023-12-16T07:28:12.402591+0000 mon.a (mon.0) 9 : audit [INF] from='client.? 172.21.15.7:0/2933925246' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-16T07:28:12.982 INFO:teuthology.orchestra.run.smithi007.stderr:Generating new minimal ceph.conf... 2023-12-16T07:28:14.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:14 smithi007 bash[24530]: audit 2023-12-16T07:28:14.102632+0000 mon.a (mon.0) 10 : audit [DBG] from='client.? 172.21.15.7:0/1992201685' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:28:14.696 INFO:teuthology.orchestra.run.smithi007.stderr:Restarting the monitor... 2023-12-16T07:28:15.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:14 smithi007 systemd[1]: Stopping Ceph mon.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:28:15.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:14 smithi007 bash[24924]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mon.a 2023-12-16T07:28:15.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:14 smithi007 bash[24530]: debug 2023-12-16T07:28:14.803+0000 7f69545a0700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-16T07:28:15.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:14 smithi007 bash[24530]: debug 2023-12-16T07:28:14.803+0000 7f69545a0700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-16T07:28:15.698 INFO:teuthology.orchestra.run.smithi007.stderr:Setting mon public_network to 172.21.0.0/20 2023-12-16T07:28:15.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:15 smithi007 bash[24933]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mon-a 2023-12-16T07:28:15.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:15 smithi007 bash[24973]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mon.a 2023-12-16T07:28:15.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:15 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.a.service: Succeeded. 2023-12-16T07:28:15.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:15 smithi007 systemd[1]: Stopped Ceph mon.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:28:15.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:15 smithi007 systemd[1]: Started Ceph mon.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:28:17.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.923+0000 7fdcaa9d9880 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-16T07:28:17.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.923+0000 7fdcaa9d9880 0 ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 7 2023-12-16T07:28:17.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.923+0000 7fdcaa9d9880 0 pidfile_write: ignore empty --pid-file 2023-12-16T07:28:17.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 0 load: jerasure load: lrc 2023-12-16T07:28:17.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: RocksDB version: 6.15.5 2023-12-16T07:28:17.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-16T07:28:17.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Compile date Apr 18 2022 2023-12-16T07:28:17.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: DB SUMMARY 2023-12-16T07:28:17.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: DB Session ID: TGCYH6ZPSWENO6HF934U 2023-12-16T07:28:17.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: CURRENT file: CURRENT 2023-12-16T07:28:17.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: IDENTITY file: IDENTITY 2023-12-16T07:28:17.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2023-12-16T07:28:17.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-12-16T07:28:17.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82081 ; 2023-12-16T07:28:17.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.error_if_exists: 0 2023-12-16T07:28:17.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.create_if_missing: 0 2023-12-16T07:28:17.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.paranoid_checks: 1 2023-12-16T07:28:17.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-12-16T07:28:17.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.env: 0x55d1e8942860 2023-12-16T07:28:17.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.fs: Posix File System 2023-12-16T07:28:17.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.info_log: 0x55d1eada5e00 2023-12-16T07:28:17.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-16T07:28:17.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.statistics: (nil) 2023-12-16T07:28:17.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.use_fsync: 0 2023-12-16T07:28:17.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_log_file_size: 0 2023-12-16T07:28:17.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-16T07:28:17.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-16T07:28:17.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-16T07:28:17.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-16T07:28:17.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.allow_fallocate: 1 2023-12-16T07:28:17.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-16T07:28:17.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-16T07:28:17.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.use_direct_reads: 0 2023-12-16T07:28:17.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-16T07:28:17.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.create_missing_column_families: 0 2023-12-16T07:28:17.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.db_log_dir: 2023-12-16T07:28:17.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2023-12-16T07:28:17.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-16T07:28:17.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-16T07:28:17.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-16T07:28:17.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-16T07:28:17.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-16T07:28:17.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-16T07:28:17.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.advise_random_on_open: 1 2023-12-16T07:28:17.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-16T07:28:17.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.write_buffer_manager: 0x55d1eae962d0 2023-12-16T07:28:17.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-16T07:28:17.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-16T07:28:17.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-16T07:28:17.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-16T07:28:17.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.rate_limiter: (nil) 2023-12-16T07:28:17.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-16T07:28:17.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-16T07:28:17.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-16T07:28:17.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-16T07:28:17.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.unordered_write: 0 2023-12-16T07:28:17.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-16T07:28:17.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-16T07:28:17.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-16T07:28:17.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-16T07:28:17.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.row_cache: None 2023-12-16T07:28:17.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.wal_filter: None 2023-12-16T07:28:17.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-16T07:28:17.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-16T07:28:17.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.preserve_deletes: 0 2023-12-16T07:28:17.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.two_write_queues: 0 2023-12-16T07:28:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.manual_wal_flush: 0 2023-12-16T07:28:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.atomic_flush: 0 2023-12-16T07:28:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-16T07:28:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-16T07:28:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-16T07:28:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.log_readahead_size: 0 2023-12-16T07:28:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-12-16T07:28:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.best_efforts_recovery: 0 2023-12-16T07:28:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-12-16T07:28:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-12-16T07:28:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.allow_data_in_errors: 0 2023-12-16T07:28:17.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.db_host_id: __hostname__ 2023-12-16T07:28:17.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_background_jobs: 2 2023-12-16T07:28:17.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_background_compactions: -1 2023-12-16T07:28:17.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_subcompactions: 1 2023-12-16T07:28:17.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-16T07:28:17.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-16T07:28:17.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-16T07:28:17.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_total_wal_size: 0 2023-12-16T07:28:17.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-16T07:28:17.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-16T07:28:17.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-16T07:28:17.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-16T07:28:17.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_open_files: -1 2023-12-16T07:28:17.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.bytes_per_sync: 0 2023-12-16T07:28:17.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-16T07:28:17.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-16T07:28:17.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-16T07:28:17.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_background_flushes: -1 2023-12-16T07:28:17.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Compression algorithms supported: 2023-12-16T07:28:17.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-16T07:28:17.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: kZSTD supported: 0 2023-12-16T07:28:17.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: kXpressCompression supported: 0 2023-12-16T07:28:17.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-16T07:28:17.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: kLZ4Compression supported: 1 2023-12-16T07:28:17.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: kBZip2Compression supported: 0 2023-12-16T07:28:17.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: kZlibCompression supported: 1 2023-12-16T07:28:17.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: kSnappyCompression supported: 1 2023-12-16T07:28:17.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-16T07:28:17.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2023-12-16T07:28:17.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2023-12-16T07:28:17.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-16T07:28:17.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.merge_operator: 2023-12-16T07:28:17.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compaction_filter: None 2023-12-16T07:28:17.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compaction_filter_factory: None 2023-12-16T07:28:17.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.sst_partitioner_factory: None 2023-12-16T07:28:17.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-16T07:28:17.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-16T07:28:17.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d1ead73d00) 2023-12-16T07:28:17.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: cache_index_and_filter_blocks: 1 2023-12-16T07:28:17.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-16T07:28:17.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-16T07:28:17.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: pin_top_level_index_and_filter: 1 2023-12-16T07:28:17.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: index_type: 0 2023-12-16T07:28:17.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: data_block_index_type: 0 2023-12-16T07:28:17.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: index_shortening: 1 2023-12-16T07:28:17.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: data_block_hash_table_util_ratio: 0.750000 2023-12-16T07:28:17.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: hash_index_allow_collision: 1 2023-12-16T07:28:17.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: checksum: 1 2023-12-16T07:28:17.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: no_block_cache: 0 2023-12-16T07:28:17.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: block_cache: 0x55d1eadda170 2023-12-16T07:28:17.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: block_cache_name: BinnedLRUCache 2023-12-16T07:28:17.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: block_cache_options: 2023-12-16T07:28:17.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: capacity : 536870912 2023-12-16T07:28:17.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: num_shard_bits : 4 2023-12-16T07:28:17.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: strict_capacity_limit : 0 2023-12-16T07:28:17.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: high_pri_pool_ratio: 0.000 2023-12-16T07:28:17.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: block_cache_compressed: (nil) 2023-12-16T07:28:17.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: persistent_cache: (nil) 2023-12-16T07:28:17.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: block_size: 4096 2023-12-16T07:28:17.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: block_size_deviation: 10 2023-12-16T07:28:17.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: block_restart_interval: 16 2023-12-16T07:28:17.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: index_block_restart_interval: 1 2023-12-16T07:28:17.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: metadata_block_size: 4096 2023-12-16T07:28:17.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: partition_filters: 0 2023-12-16T07:28:17.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: use_delta_encoding: 1 2023-12-16T07:28:17.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-16T07:28:17.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: whole_key_filtering: 1 2023-12-16T07:28:17.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: verify_compression: 0 2023-12-16T07:28:17.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: read_amp_bytes_per_bit: 0 2023-12-16T07:28:17.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: format_version: 4 2023-12-16T07:28:17.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: enable_index_compression: 1 2023-12-16T07:28:17.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: block_align: 0 2023-12-16T07:28:17.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-16T07:28:17.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-16T07:28:17.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compression: NoCompression 2023-12-16T07:28:17.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-16T07:28:17.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-16T07:28:17.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-16T07:28:17.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.num_levels: 7 2023-12-16T07:28:17.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-16T07:28:17.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-16T07:28:17.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-16T07:28:17.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-16T07:28:17.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-16T07:28:17.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-16T07:28:17.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-16T07:28:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-16T07:28:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-12-16T07:28:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-16T07:28:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-16T07:28:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compression_opts.level: 32767 2023-12-16T07:28:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-16T07:28:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-16T07:28:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-16T07:28:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-12-16T07:28:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compression_opts.enabled: false 2023-12-16T07:28:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-16T07:28:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-16T07:28:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-16T07:28:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-16T07:28:17.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-16T07:28:17.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-16T07:28:17.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-16T07:28:17.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-16T07:28:17.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-16T07:28:17.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-16T07:28:17.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-16T07:28:17.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-16T07:28:17.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-16T07:28:17.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-16T07:28:17.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-16T07:28:17.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-16T07:28:17.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-16T07:28:17.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.arena_block_size: 4194304 2023-12-16T07:28:17.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-16T07:28:17.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-16T07:28:17.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-16T07:28:17.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-16T07:28:17.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-16T07:28:17.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-16T07:28:17.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-16T07:28:17.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-16T07:28:17.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-16T07:28:17.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-16T07:28:17.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-16T07:28:17.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-16T07:28:17.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-16T07:28:17.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-16T07:28:17.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.table_properties_collectors: 2023-12-16T07:28:17.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.inplace_update_support: 0 2023-12-16T07:28:17.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-16T07:28:17.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-16T07:28:17.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-16T07:28:17.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-16T07:28:17.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.bloom_locality: 0 2023-12-16T07:28:17.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.max_successive_merges: 0 2023-12-16T07:28:17.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-16T07:28:17.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-16T07:28:17.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.force_consistency_checks: 1 2023-12-16T07:28:17.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-16T07:28:17.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.ttl: 2592000 2023-12-16T07:28:17.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-16T07:28:17.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.enable_blob_files: false 2023-12-16T07:28:17.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.min_blob_size: 0 2023-12-16T07:28:17.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.blob_file_size: 268435456 2023-12-16T07:28:17.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.blob_compression_type: NoCompression 2023-12-16T07:28:17.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-12-16T07:28:17.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-12-16T07:28:17.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-16T07:28:17.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2023-12-16T07:28:17.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: [db/version_set.cc:4083] Creating manifest 13 2023-12-16T07:28:17.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702711696937584, "job": 1, "event": "recovery_started", "wal_files": [10]} 2023-12-16T07:28:17.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 4 rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2023-12-16T07:28:17.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.931+0000 7fdcaa9d9880 3 rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-16T07:28:17.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.935+0000 7fdcaa9d9880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702711696938736, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 78790, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77053, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9896, "raw_average_key_size": 47, "raw_value_size": 71324, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 209, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1702711696, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "efa1eb90-5256-4a90-8760-5f545d6f2ce7", "db_session_id": "TGCYH6ZPSWENO6HF934U"}} 2023-12-16T07:28:17.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.935+0000 7fdcaa9d9880 4 rocksdb: [db/version_set.cc:4083] Creating manifest 15 2023-12-16T07:28:17.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.935+0000 7fdcaa9d9880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702711696939762, "job": 1, "event": "recovery_finished"} 2023-12-16T07:28:17.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.935+0000 7fdcaa9d9880 4 rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-12-16T07:28:17.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.935+0000 7fdcaa9d9880 4 rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55d1eadc0a80 2023-12-16T07:28:17.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.935+0000 7fdcaa9d9880 4 rocksdb: DB pointer 0x55d1eadd0000 2023-12-16T07:28:17.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.935+0000 7fdc9a761700 4 rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2023-12-16T07:28:17.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.935+0000 7fdc9a761700 4 rocksdb: [db/db_impl/db_impl.cc:903] 2023-12-16T07:28:17.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: ** DB Stats ** 2023-12-16T07:28:17.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:28:17.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-16T07:28:17.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-16T07:28:17.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T07:28:17.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-16T07:28:17.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-16T07:28:17.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T07:28:17.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: ** Compaction Stats [default] ** 2023-12-16T07:28:17.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:28:17.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:28:17.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: L0 2/0 78.70 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 93.5 0.00 0.00 1 0.001 0 0 2023-12-16T07:28:17.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Sum 2/0 78.70 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 93.5 0.00 0.00 1 0.001 0 0 2023-12-16T07:28:17.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 93.5 0.00 0.00 1 0.001 0 0 2023-12-16T07:28:17.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: ** Compaction Stats [default] ** 2023-12-16T07:28:17.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:28:17.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:28:17.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 93.5 0.00 0.00 1 0.001 0 0 2023-12-16T07:28:17.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:28:17.092 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T07:28:17.092 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T07:28:17.092 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T07:28:17.092 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T07:28:17.092 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: AddFile(Keys): cumulative 0, interval 0 2023-12-16T07:28:17.092 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Cumulative compaction: 0.00 GB write, 14.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:28:17.092 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Interval compaction: 0.00 GB write, 14.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:28:17.092 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T07:28:17.092 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: ** File Read Latency Histogram By Level [default] ** 2023-12-16T07:28:17.092 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: ** Compaction Stats [default] ** 2023-12-16T07:28:17.092 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:28:17.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:28:17.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: L0 2/0 78.70 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 93.5 0.00 0.00 1 0.001 0 0 2023-12-16T07:28:17.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Sum 2/0 78.70 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 93.5 0.00 0.00 1 0.001 0 0 2023-12-16T07:28:17.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-16T07:28:17.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: ** Compaction Stats [default] ** 2023-12-16T07:28:17.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:28:17.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:28:17.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 93.5 0.00 0.00 1 0.001 0 0 2023-12-16T07:28:17.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:28:17.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T07:28:17.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T07:28:17.093 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T07:28:17.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T07:28:17.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: AddFile(Keys): cumulative 0, interval 0 2023-12-16T07:28:17.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Cumulative compaction: 0.00 GB write, 14.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:28:17.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:28:17.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T07:28:17.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: ** File Read Latency Histogram By Level [default] ** 2023-12-16T07:28:17.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.935+0000 7fdcaa9d9880 0 starting mon.a rank 0 at public addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] at bind addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:28:17.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 1 mon.a@-1(???) e1 preinit fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:28:17.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 0 mon.a@-1(???).mds e1 new map 2023-12-16T07:28:17.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 0 mon.a@-1(???).mds e1 print_map 2023-12-16T07:28:17.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: e1 2023-12-16T07:28:17.094 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: enable_multiple, ever_enabled_multiple: 1,1 2023-12-16T07:28:17.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T07:28:17.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: legacy client fscid: -1 2023-12-16T07:28:17.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: 2023-12-16T07:28:17.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: No filesystems configured 2023-12-16T07:28:17.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-16T07:28:17.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T07:28:17.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T07:28:17.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T07:28:17.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-12-16T07:28:17.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 20 expand_channel_meta expand map: {default=false} 2023-12-16T07:28:17.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 20 expand_channel_meta from 'false' to 'false' 2023-12-16T07:28:17.095 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 20 expand_channel_meta expanded map: {default=false} 2023-12-16T07:28:17.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 20 expand_channel_meta expand map: {default=info} 2023-12-16T07:28:17.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 20 expand_channel_meta from 'info' to 'info' 2023-12-16T07:28:17.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 20 expand_channel_meta expanded map: {default=info} 2023-12-16T07:28:17.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 20 expand_channel_meta expand map: {default=daemon} 2023-12-16T07:28:17.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 20 expand_channel_meta from 'daemon' to 'daemon' 2023-12-16T07:28:17.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 20 expand_channel_meta expanded map: {default=daemon} 2023-12-16T07:28:17.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 20 expand_channel_meta expand map: {default=debug} 2023-12-16T07:28:17.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 20 expand_channel_meta from 'debug' to 'debug' 2023-12-16T07:28:17.096 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:16 smithi007 bash[25009]: debug 2023-12-16T07:28:16.939+0000 7fdcaa9d9880 20 expand_channel_meta expanded map: {default=debug} 2023-12-16T07:28:17.816 INFO:teuthology.orchestra.run.smithi007.stderr:Wrote config to /etc/ceph/ceph.conf 2023-12-16T07:28:17.816 INFO:teuthology.orchestra.run.smithi007.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-12-16T07:28:17.816 INFO:teuthology.orchestra.run.smithi007.stderr:Creating mgr... 2023-12-16T07:28:17.816 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying port 9283 ... 2023-12-16T07:28:18.040 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to reset failed state of unit ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.y.service: Unit ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.y.service not loaded. 2023-12-16T07:28:18.047 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715.target.wants/ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.y.service -> /etc/systemd/system/ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@.service. 2023-12-16T07:28:18.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:17 smithi007 bash[25009]: cluster 2023-12-16T07:28:16.947764+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2023-12-16T07:28:18.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:17 smithi007 bash[25009]: cluster 2023-12-16T07:28:16.947851+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]} 2023-12-16T07:28:18.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:17 smithi007 bash[25009]: cluster 2023-12-16T07:28:16.947978+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2023-12-16T07:28:18.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:17 smithi007 bash[25009]: cluster 2023-12-16T07:28:16.948032+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-16T07:28:18.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:17 smithi007 bash[25009]: cluster 2023-12-16T07:28:16.948458+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-12-16T07:28:18.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:17 smithi007 bash[25009]: audit 2023-12-16T07:28:17.213979+0000 mon.a (mon.0) 6 : audit [INF] from='client.? 172.21.15.7:0/1359183701' entity='client.admin' 2023-12-16T07:28:18.253 INFO:teuthology.orchestra.run.smithi007.stderr:firewalld does not appear to be present 2023-12-16T07:28:18.253 INFO:teuthology.orchestra.run.smithi007.stderr:Not possible to enable service . firewalld.service is not available 2023-12-16T07:28:18.254 INFO:teuthology.orchestra.run.smithi007.stderr:firewalld does not appear to be present 2023-12-16T07:28:18.254 INFO:teuthology.orchestra.run.smithi007.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-12-16T07:28:18.254 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mgr to start... 2023-12-16T07:28:18.254 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mgr... 2023-12-16T07:28:19.852 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2023-12-16T07:28:19.852 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2023-12-16T07:28:19.853 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsid": "8b71d3dc-9be4-11ee-95a5-87774f69a715", 2023-12-16T07:28:19.853 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "health": { 2023-12-16T07:28:19.853 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-16T07:28:19.853 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "checks": {}, 2023-12-16T07:28:19.853 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mutes": [] 2023-12-16T07:28:19.853 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:19.853 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-16T07:28:19.853 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum": [ 2023-12-16T07:28:19.853 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 0 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "a" 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_age": 2, 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "monmap": { 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osdmap": { 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-16T07:28:19.854 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-16T07:28:19.855 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-16T07:28:19.855 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-16T07:28:19.855 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-16T07:28:19.858 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:19.858 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgmap": { 2023-12-16T07:28:19.858 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-16T07:28:19.858 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-16T07:28:19.858 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-16T07:28:19.858 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-16T07:28:19.858 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-16T07:28:19.858 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-16T07:28:19.858 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-16T07:28:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-16T07:28:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsmap": { 2023-12-16T07:28:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-16T07:28:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-16T07:28:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-16T07:28:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": false, 2023-12-16T07:28:19.859 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-16T07:28:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modules": [ 2023-12-16T07:28:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "iostat", 2023-12-16T07:28:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "nfs", 2023-12-16T07:28:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "restful" 2023-12-16T07:28:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2023-12-16T07:28:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2023-12-16T07:28:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "servicemap": { 2023-12-16T07:28:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modified": "2023-12-16T07:28:10.359720+0000", 2023-12-16T07:28:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2023-12-16T07:28:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-16T07:28:19.860 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2023-12-16T07:28:20.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:19 smithi007 bash[25009]: audit 2023-12-16T07:28:19.848981+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.7:0/3334983161' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-16T07:28:20.065 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:19 smithi007 bash[25282]: debug 2023-12-16T07:28:19.751+0000 7f76dc974000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T07:28:20.448 INFO:teuthology.orchestra.run.smithi007.stderr:mgr not available, waiting (1/15)... 2023-12-16T07:28:20.925 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:20 smithi007 bash[25282]: debug 2023-12-16T07:28:20.631+0000 7f76dc974000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T07:28:20.925 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:20 smithi007 bash[25282]: debug 2023-12-16T07:28:20.755+0000 7f76dc974000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T07:28:21.261 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:20 smithi007 bash[25282]: debug 2023-12-16T07:28:20.919+0000 7f76dc974000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:28:21.564 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:21 smithi007 bash[25282]: debug 2023-12-16T07:28:21.255+0000 7f76dc974000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T07:28:21.564 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:21 smithi007 bash[25282]: debug 2023-12-16T07:28:21.351+0000 7f76dc974000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T07:28:21.815 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:21 smithi007 bash[25282]: debug 2023-12-16T07:28:21.559+0000 7f76dc974000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T07:28:22.771 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:22 smithi007 bash[25282]: debug 2023-12-16T07:28:22.447+0000 7f76dc974000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T07:28:22.772 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:22 smithi007 bash[25282]: debug 2023-12-16T07:28:22.559+0000 7f76dc974000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T07:28:23.065 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:22 smithi007 bash[25282]: debug 2023-12-16T07:28:22.767+0000 7f76dc974000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T07:28:23.436 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:23 smithi007 bash[25282]: debug 2023-12-16T07:28:23.127+0000 7f76dc974000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T07:28:23.736 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2023-12-16T07:28:23.737 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2023-12-16T07:28:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsid": "8b71d3dc-9be4-11ee-95a5-87774f69a715", 2023-12-16T07:28:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "health": { 2023-12-16T07:28:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-16T07:28:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "checks": {}, 2023-12-16T07:28:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mutes": [] 2023-12-16T07:28:23.738 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:23.739 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-16T07:28:23.739 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum": [ 2023-12-16T07:28:23.739 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 0 2023-12-16T07:28:23.739 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2023-12-16T07:28:23.740 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-16T07:28:23.740 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "a" 2023-12-16T07:28:23.740 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2023-12-16T07:28:23.740 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_age": 6, 2023-12-16T07:28:23.741 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "monmap": { 2023-12-16T07:28:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2023-12-16T07:28:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-16T07:28:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osdmap": { 2023-12-16T07:28:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-16T07:28:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-16T07:28:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-16T07:28:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-16T07:28:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-16T07:28:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-16T07:28:23.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:23.744 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgmap": { 2023-12-16T07:28:23.744 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-16T07:28:23.744 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-16T07:28:23.745 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-16T07:28:23.745 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-16T07:28:23.745 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-16T07:28:23.745 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-16T07:28:23.745 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-16T07:28:23.745 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-16T07:28:23.745 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:23.745 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsmap": { 2023-12-16T07:28:23.745 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:23.745 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-16T07:28:23.745 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-16T07:28:23.745 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:23.745 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-16T07:28:23.745 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": false, 2023-12-16T07:28:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-16T07:28:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modules": [ 2023-12-16T07:28:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "iostat", 2023-12-16T07:28:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "nfs", 2023-12-16T07:28:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "restful" 2023-12-16T07:28:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2023-12-16T07:28:23.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2023-12-16T07:28:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "servicemap": { 2023-12-16T07:28:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modified": "2023-12-16T07:28:10.359720+0000", 2023-12-16T07:28:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2023-12-16T07:28:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-16T07:28:23.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2023-12-16T07:28:23.760 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:23 smithi007 bash[25282]: debug 2023-12-16T07:28:23.463+0000 7f76dc974000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T07:28:24.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:23 smithi007 bash[25009]: audit 2023-12-16T07:28:23.735676+0000 mon.a (mon.0) 8 : audit [DBG] from='client.? 172.21.15.7:0/1795414628' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-16T07:28:24.065 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:23 smithi007 bash[25282]: debug 2023-12-16T07:28:23.843+0000 7f76dc974000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T07:28:24.315 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:24 smithi007 bash[25282]: debug 2023-12-16T07:28:24.119+0000 7f76dc974000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:28:24.320 INFO:teuthology.orchestra.run.smithi007.stderr:mgr not available, waiting (2/15)... 2023-12-16T07:28:25.096 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:24 smithi007 bash[25282]: debug 2023-12-16T07:28:24.823+0000 7f76dc974000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T07:28:25.096 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:24 smithi007 bash[25282]: debug 2023-12-16T07:28:24.927+0000 7f76dc974000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T07:28:25.347 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:25 smithi007 bash[25282]: debug 2023-12-16T07:28:25.091+0000 7f76dc974000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T07:28:25.347 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:25 smithi007 bash[25282]: debug 2023-12-16T07:28:25.199+0000 7f76dc974000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T07:28:25.645 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:25 smithi007 bash[25282]: debug 2023-12-16T07:28:25.343+0000 7f76dc974000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T07:28:25.950 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:25 smithi007 bash[25282]: debug 2023-12-16T07:28:25.635+0000 7f76dc974000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T07:28:25.950 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:25 smithi007 bash[25282]: debug 2023-12-16T07:28:25.759+0000 7f76dc974000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T07:28:26.315 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:25 smithi007 bash[25282]: debug 2023-12-16T07:28:25.943+0000 7f76dc974000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T07:28:26.315 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:26 smithi007 bash[25282]: debug 2023-12-16T07:28:26.067+0000 7f76dc974000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T07:28:27.315 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:26 smithi007 bash[25282]: debug 2023-12-16T07:28:26.942+0000 7f76dc974000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T07:28:27.448 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2023-12-16T07:28:27.448 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2023-12-16T07:28:27.448 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsid": "8b71d3dc-9be4-11ee-95a5-87774f69a715", 2023-12-16T07:28:27.448 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "health": { 2023-12-16T07:28:27.448 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-16T07:28:27.448 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "checks": {}, 2023-12-16T07:28:27.448 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mutes": [] 2023-12-16T07:28:27.448 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum": [ 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 0 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "a" 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_age": 10, 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "monmap": { 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osdmap": { 2023-12-16T07:28:27.449 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgmap": { 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-16T07:28:27.450 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsmap": { 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": false, 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modules": [ 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "iostat", 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "nfs", 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "restful" 2023-12-16T07:28:27.451 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2023-12-16T07:28:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2023-12-16T07:28:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "servicemap": { 2023-12-16T07:28:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modified": "2023-12-16T07:28:10.359720+0000", 2023-12-16T07:28:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2023-12-16T07:28:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-16T07:28:27.452 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2023-12-16T07:28:28.085 INFO:teuthology.orchestra.run.smithi007.stderr:mgr not available, waiting (3/15)... 2023-12-16T07:28:28.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:27 smithi007 bash[25009]: cluster 2023-12-16T07:28:26.968807+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2023-12-16T07:28:28.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:27 smithi007 bash[25009]: cluster 2023-12-16T07:28:26.974731+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00619723s) 2023-12-16T07:28:28.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:27 smithi007 bash[25009]: audit 2023-12-16T07:28:26.976643+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.7:0/866806949' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:28:28.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:27 smithi007 bash[25009]: audit 2023-12-16T07:28:26.976956+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.7:0/866806949' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:28:28.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:27 smithi007 bash[25009]: audit 2023-12-16T07:28:26.977237+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.7:0/866806949' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:28:28.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:27 smithi007 bash[25009]: audit 2023-12-16T07:28:26.977613+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.7:0/866806949' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:28:28.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:27 smithi007 bash[25009]: audit 2023-12-16T07:28:26.978775+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.7:0/866806949' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:28:28.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:27 smithi007 bash[25009]: cluster 2023-12-16T07:28:26.990266+0000 mon.a (mon.0) 16 : cluster [INF] Manager daemon y is now available 2023-12-16T07:28:28.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:27 smithi007 bash[25009]: audit 2023-12-16T07:28:27.011262+0000 mon.a (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.7:0/866806949' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:28:28.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:27 smithi007 bash[25009]: audit 2023-12-16T07:28:27.013722+0000 mon.a (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.7:0/866806949' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:28:28.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:27 smithi007 bash[25009]: audit 2023-12-16T07:28:27.025626+0000 mon.a (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.7:0/866806949' entity='mgr.y' 2023-12-16T07:28:28.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:27 smithi007 bash[25009]: audit 2023-12-16T07:28:27.029773+0000 mon.a (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.7:0/866806949' entity='mgr.y' 2023-12-16T07:28:28.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:27 smithi007 bash[25009]: audit 2023-12-16T07:28:27.033901+0000 mon.a (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.7:0/866806949' entity='mgr.y' 2023-12-16T07:28:28.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:27 smithi007 bash[25009]: audit 2023-12-16T07:28:27.442424+0000 mon.a (mon.0) 22 : audit [DBG] from='client.? 172.21.15.7:0/3516817756' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-16T07:28:29.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:28 smithi007 bash[25009]: cluster 2023-12-16T07:28:27.983060+0000 mon.a (mon.0) 23 : cluster [DBG] mgrmap e3: y(active, since 1.01452s) 2023-12-16T07:28:31.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:30 smithi007 bash[25009]: cluster 2023-12-16T07:28:29.981795+0000 mon.a (mon.0) 24 : cluster [DBG] mgrmap e4: y(active, since 3s) 2023-12-16T07:28:31.610 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2023-12-16T07:28:31.610 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2023-12-16T07:28:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsid": "8b71d3dc-9be4-11ee-95a5-87774f69a715", 2023-12-16T07:28:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "health": { 2023-12-16T07:28:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-16T07:28:31.611 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "checks": {}, 2023-12-16T07:28:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mutes": [] 2023-12-16T07:28:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:31.612 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-16T07:28:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum": [ 2023-12-16T07:28:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 0 2023-12-16T07:28:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2023-12-16T07:28:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-16T07:28:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "a" 2023-12-16T07:28:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2023-12-16T07:28:31.613 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_age": 14, 2023-12-16T07:28:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "monmap": { 2023-12-16T07:28:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2023-12-16T07:28:31.614 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-16T07:28:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osdmap": { 2023-12-16T07:28:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-16T07:28:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-16T07:28:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-16T07:28:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-16T07:28:31.615 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-16T07:28:31.616 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-16T07:28:31.616 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgmap": { 2023-12-16T07:28:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-16T07:28:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-16T07:28:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-16T07:28:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-16T07:28:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-16T07:28:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-16T07:28:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-16T07:28:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-16T07:28:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:31.618 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsmap": { 2023-12-16T07:28:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-16T07:28:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-16T07:28:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-16T07:28:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": true, 2023-12-16T07:28:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-16T07:28:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modules": [ 2023-12-16T07:28:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "iostat", 2023-12-16T07:28:31.619 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "nfs", 2023-12-16T07:28:31.620 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "restful" 2023-12-16T07:28:31.620 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2023-12-16T07:28:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2023-12-16T07:28:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "servicemap": { 2023-12-16T07:28:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-16T07:28:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modified": "2023-12-16T07:28:10.359720+0000", 2023-12-16T07:28:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2023-12-16T07:28:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2023-12-16T07:28:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-16T07:28:31.621 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2023-12-16T07:28:32.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:31 smithi007 bash[25009]: audit 2023-12-16T07:28:31.609278+0000 mon.a (mon.0) 25 : audit [DBG] from='client.? 172.21.15.7:0/3907874708' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-16T07:28:32.215 INFO:teuthology.orchestra.run.smithi007.stderr:mgr is available 2023-12-16T07:28:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2023-12-16T07:28:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [global] 2023-12-16T07:28:33.576 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: fsid = 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:28:33.577 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-12-16T07:28:33.577 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-12-16T07:28:33.577 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-12-16T07:28:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-12-16T07:28:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2023-12-16T07:28:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [mgr] 2023-12-16T07:28:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-12-16T07:28:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2023-12-16T07:28:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [osd] 2023-12-16T07:28:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-12-16T07:28:33.578 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-12-16T07:28:34.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:33 smithi007 bash[25009]: audit 2023-12-16T07:28:33.570024+0000 mon.a (mon.0) 26 : audit [INF] from='client.? 172.21.15.7:0/3284992345' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-16T07:28:34.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:33 smithi007 bash[25009]: audit 2023-12-16T07:28:33.574838+0000 mon.a (mon.0) 27 : audit [INF] from='client.? 172.21.15.7:0/3284992345' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-16T07:28:34.172 INFO:teuthology.orchestra.run.smithi007.stderr:Enabling cephadm module... 2023-12-16T07:28:36.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:35 smithi007 bash[25009]: audit 2023-12-16T07:28:35.575662+0000 mon.a (mon.0) 28 : audit [INF] from='client.? 172.21.15.7:0/2288996752' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-12-16T07:28:36.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:35 smithi007 bash[25282]: ignoring --setuser ceph since I am not root 2023-12-16T07:28:36.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:35 smithi007 bash[25282]: ignoring --setgroup ceph since I am not root 2023-12-16T07:28:36.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:35 smithi007 bash[25282]: debug 2023-12-16T07:28:35.830+0000 7f6675820000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T07:28:36.815 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:36 smithi007 bash[25282]: debug 2023-12-16T07:28:36.522+0000 7f6675820000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T07:28:36.815 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:36 smithi007 bash[25282]: debug 2023-12-16T07:28:36.646+0000 7f6675820000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T07:28:36.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:36 smithi007 bash[25009]: audit 2023-12-16T07:28:35.633222+0000 mon.a (mon.0) 29 : audit [INF] from='client.? 172.21.15.7:0/2288996752' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-12-16T07:28:36.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:36 smithi007 bash[25009]: cluster 2023-12-16T07:28:35.633404+0000 mon.a (mon.0) 30 : cluster [DBG] mgrmap e5: y(active, since 8s) 2023-12-16T07:28:37.175 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:36 smithi007 bash[25282]: debug 2023-12-16T07:28:36.814+0000 7f6675820000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:28:37.507 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:37 smithi007 bash[25282]: debug 2023-12-16T07:28:37.170+0000 7f6675820000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T07:28:37.507 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:37 smithi007 bash[25282]: debug 2023-12-16T07:28:37.278+0000 7f6675820000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T07:28:37.653 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2023-12-16T07:28:37.653 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 5, 2023-12-16T07:28:37.653 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": true, 2023-12-16T07:28:37.654 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "active_name": "y", 2023-12-16T07:28:37.654 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-16T07:28:37.654 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2023-12-16T07:28:37.805 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:37 smithi007 bash[25282]: debug 2023-12-16T07:28:37.502+0000 7f6675820000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T07:28:38.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:37 smithi007 bash[25009]: audit 2023-12-16T07:28:37.652109+0000 mon.a (mon.0) 31 : audit [DBG] from='client.? 172.21.15.7:0/1403514324' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-16T07:28:38.379 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for the mgr to restart... 2023-12-16T07:28:38.379 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mgr epoch 5... 2023-12-16T07:28:38.728 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:38 smithi007 bash[25282]: debug 2023-12-16T07:28:38.402+0000 7f6675820000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T07:28:38.728 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:38 smithi007 bash[25282]: debug 2023-12-16T07:28:38.510+0000 7f6675820000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T07:28:39.065 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:38 smithi007 bash[25282]: debug 2023-12-16T07:28:38.722+0000 7f6675820000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T07:28:39.436 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:39 smithi007 bash[25282]: debug 2023-12-16T07:28:39.086+0000 7f6675820000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T07:28:39.815 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:39 smithi007 bash[25282]: debug 2023-12-16T07:28:39.430+0000 7f6675820000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T07:28:40.103 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:39 smithi007 bash[25282]: debug 2023-12-16T07:28:39.814+0000 7f6675820000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T07:28:40.565 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:40 smithi007 bash[25282]: debug 2023-12-16T07:28:40.098+0000 7f6675820000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:28:41.065 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:40 smithi007 bash[25282]: debug 2023-12-16T07:28:40.802+0000 7f6675820000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T07:28:41.065 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:40 smithi007 bash[25282]: debug 2023-12-16T07:28:40.906+0000 7f6675820000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T07:28:41.330 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:41 smithi007 bash[25282]: debug 2023-12-16T07:28:41.070+0000 7f6675820000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T07:28:41.331 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:41 smithi007 bash[25282]: debug 2023-12-16T07:28:41.182+0000 7f6675820000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T07:28:41.627 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:41 smithi007 bash[25282]: debug 2023-12-16T07:28:41.322+0000 7f6675820000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T07:28:41.936 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:41 smithi007 bash[25282]: debug 2023-12-16T07:28:41.618+0000 7f6675820000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T07:28:41.936 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:41 smithi007 bash[25282]: debug 2023-12-16T07:28:41.742+0000 7f6675820000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T07:28:42.315 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:41 smithi007 bash[25282]: debug 2023-12-16T07:28:41.930+0000 7f6675820000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T07:28:42.315 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:42 smithi007 bash[25282]: debug 2023-12-16T07:28:42.050+0000 7f6675820000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T07:28:43.316 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:42 smithi007 bash[25282]: debug 2023-12-16T07:28:42.930+0000 7f6675820000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T07:28:44.032 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2023-12-16T07:28:44.032 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-12-16T07:28:44.032 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "initialized": true 2023-12-16T07:28:44.032 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2023-12-16T07:28:44.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: cluster 2023-12-16T07:28:42.956720+0000 mon.a (mon.0) 32 : cluster [INF] Active manager daemon y restarted 2023-12-16T07:28:44.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: cluster 2023-12-16T07:28:42.957542+0000 mon.a (mon.0) 33 : cluster [INF] Activating manager daemon y 2023-12-16T07:28:44.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: cluster 2023-12-16T07:28:42.961056+0000 mon.a (mon.0) 34 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-12-16T07:28:44.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: cluster 2023-12-16T07:28:43.014081+0000 mon.a (mon.0) 35 : cluster [DBG] mgrmap e6: y(active, starting, since 0.0567032s) 2023-12-16T07:28:44.238 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: audit 2023-12-16T07:28:43.016538+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:28:44.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: audit 2023-12-16T07:28:43.018903+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:28:44.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: audit 2023-12-16T07:28:43.020632+0000 mon.a (mon.0) 38 : audit [DBG] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:28:44.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: audit 2023-12-16T07:28:43.021014+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:28:44.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: audit 2023-12-16T07:28:43.021263+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:28:44.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: cluster 2023-12-16T07:28:43.031289+0000 mon.a (mon.0) 41 : cluster [INF] Manager daemon y is now available 2023-12-16T07:28:44.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: audit 2023-12-16T07:28:43.053711+0000 mon.a (mon.0) 42 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' 2023-12-16T07:28:44.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: audit 2023-12-16T07:28:43.058765+0000 mon.a (mon.0) 43 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' 2023-12-16T07:28:44.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: audit 2023-12-16T07:28:43.064398+0000 mon.a (mon.0) 44 : audit [DBG] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:28:44.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: audit 2023-12-16T07:28:43.068152+0000 mon.a (mon.0) 45 : audit [DBG] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:28:44.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: audit 2023-12-16T07:28:43.076662+0000 mon.a (mon.0) 46 : audit [DBG] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:28:44.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: audit 2023-12-16T07:28:43.090933+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:28:44.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:43 smithi007 bash[25009]: audit 2023-12-16T07:28:43.093495+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:28:44.565 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:44 smithi007 bash[25282]: [16/Dec/2023:07:28:44] ENGINE Bus STARTING 2023-12-16T07:28:44.565 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:44 smithi007 bash[25282]: [16/Dec/2023:07:28:44] ENGINE Serving on https://172.21.15.7:7150 2023-12-16T07:28:44.565 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:44 smithi007 bash[25282]: [16/Dec/2023:07:28:44] ENGINE Bus STARTED 2023-12-16T07:28:44.706 INFO:teuthology.orchestra.run.smithi007.stderr:mgr epoch 5 is available 2023-12-16T07:28:44.706 INFO:teuthology.orchestra.run.smithi007.stderr:Setting orchestrator backend to cephadm... 2023-12-16T07:28:45.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:45 smithi007 bash[25009]: cluster 2023-12-16T07:28:44.019877+0000 mon.a (mon.0) 49 : cluster [DBG] mgrmap e7: y(active, since 1.06251s) 2023-12-16T07:28:45.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:45 smithi007 bash[25009]: cephadm 2023-12-16T07:28:44.312124+0000 mgr.y (mgr.14120) 3 : cephadm [INF] [16/Dec/2023:07:28:44] ENGINE Bus STARTING 2023-12-16T07:28:45.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:45 smithi007 bash[25009]: cephadm 2023-12-16T07:28:44.439857+0000 mgr.y (mgr.14120) 4 : cephadm [INF] [16/Dec/2023:07:28:44] ENGINE Serving on https://172.21.15.7:7150 2023-12-16T07:28:45.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:45 smithi007 bash[25009]: cephadm 2023-12-16T07:28:44.440292+0000 mgr.y (mgr.14120) 5 : cephadm [INF] [16/Dec/2023:07:28:44] ENGINE Bus STARTED 2023-12-16T07:28:45.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:45 smithi007 bash[25009]: audit 2023-12-16T07:28:44.446881+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' 2023-12-16T07:28:45.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:45 smithi007 bash[25009]: audit 2023-12-16T07:28:44.448497+0000 mon.a (mon.0) 51 : audit [DBG] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:28:46.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:46 smithi007 bash[25009]: cluster 2023-12-16T07:28:45.450994+0000 mon.a (mon.0) 52 : cluster [DBG] mgrmap e8: y(active, since 2s) 2023-12-16T07:28:46.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:46 smithi007 bash[25009]: audit 2023-12-16T07:28:46.103952+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' 2023-12-16T07:28:46.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:46 smithi007 bash[25009]: audit 2023-12-16T07:28:46.119664+0000 mon.a (mon.0) 54 : audit [DBG] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:28:47.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:47 smithi007 bash[25009]: audit 2023-12-16T07:28:46.093642+0000 mgr.y (mgr.14120) 6 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:28:48.059 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: value unchanged 2023-12-16T07:28:48.627 INFO:teuthology.orchestra.run.smithi007.stderr:Generating ssh key... 2023-12-16T07:28:49.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:48 smithi007 bash[25009]: audit 2023-12-16T07:28:48.058373+0000 mgr.y (mgr.14120) 7 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:28:50.566 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: Generating public/private rsa key pair. 2023-12-16T07:28:50.566 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: Your identification has been saved in /tmp/tmp3ded5550/key. 2023-12-16T07:28:50.566 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: Your public key has been saved in /tmp/tmp3ded5550/key.pub. 2023-12-16T07:28:50.566 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: The key fingerprint is: 2023-12-16T07:28:50.566 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: SHA256:AZT0mCOkpvhTVMbgzS8QVtNj15O2Y/zMPihPYqCKt/w ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:28:50.566 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: The key's randomart image is: 2023-12-16T07:28:50.566 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: +---[RSA 3072]----+ 2023-12-16T07:28:50.566 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: | =*Oo . . | 2023-12-16T07:28:50.566 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: | = *o*+ . = | 2023-12-16T07:28:50.566 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: | o = *.oo o o | 2023-12-16T07:28:50.566 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: |.o . o o . = | 2023-12-16T07:28:50.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: |o . . S . = | 2023-12-16T07:28:50.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: | . . o . + | 2023-12-16T07:28:50.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: | o . o .o | 2023-12-16T07:28:50.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: | +.. ..o. o | 2023-12-16T07:28:50.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: | ..+oE o. . | 2023-12-16T07:28:50.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:28:50 smithi007 bash[25282]: +----[SHA256]-----+ 2023-12-16T07:28:51.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:51 smithi007 bash[25009]: audit 2023-12-16T07:28:49.997075+0000 mgr.y (mgr.14120) 8 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:28:51.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:51 smithi007 bash[25009]: cephadm 2023-12-16T07:28:49.997505+0000 mgr.y (mgr.14120) 9 : cephadm [INF] Generating ssh key... 2023-12-16T07:28:51.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:51 smithi007 bash[25009]: audit 2023-12-16T07:28:50.280473+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' 2023-12-16T07:28:51.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:51 smithi007 bash[25009]: audit 2023-12-16T07:28:50.284175+0000 mon.a (mon.0) 56 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' 2023-12-16T07:28:52.188 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0in26ZtYrIbvbKvwXM8aK4lBMIBt2uayQJnDbfmcckmnd6d5HLtHOSCypMABi69gm8xZs/yaMGKXFnBx29Dl085B7IgzGWBnUY6p6PZhfh56QC8K6ZJcsrX+ov1Km3wiMg19uUjT1y0AWXEemUz3jBGUHc88EZfsxj71RyF8Svd7aQuCg0u48um9XxqmmsiQ1z82C6AkMoND99ucriAvV7/BhV6WC2CQ6a3uf5ap/EZkc1zhgv+26csHzVAQObGTmC0p3KeuUcnSXXARnhC38kq3RAzgkas+yeUdpI3D3OfvJjd4FDzABSUxbVIo/+BNOd5XblRaf7ivKyboOZ/36kX9rdDhy1wSAE1VlA/Qo21vLEV5j403jJ3rZv75kGkiHgGGitdR8ZQ9dOhbg1ps1dciC0AI1rHLRgf8fQJ7MaleNerJ/zCpnnCCxQcNeK+T+UoQag6nI3BUNCTMGIs+GNTxtaX+h5NtisssyD2VWZIAT/Gay8Rati56Ay59o87s= ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:28:52.755 INFO:teuthology.orchestra.run.smithi007.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-12-16T07:28:52.755 INFO:teuthology.orchestra.run.smithi007.stderr:Adding key to root@localhost authorized_keys... 2023-12-16T07:28:52.755 INFO:teuthology.orchestra.run.smithi007.stderr:Adding host smithi007... 2023-12-16T07:28:53.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:52 smithi007 bash[25009]: audit 2023-12-16T07:28:52.186899+0000 mgr.y (mgr.14120) 10 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:28:55.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:54 smithi007 bash[25009]: audit 2023-12-16T07:28:54.047862+0000 mgr.y (mgr.14120) 11 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi007", "addr": "172.21.15.7", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:28:55.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:54 smithi007 bash[25009]: cephadm 2023-12-16T07:28:54.454005+0000 mgr.y (mgr.14120) 12 : cephadm [INF] Deploying cephadm binary to smithi007 2023-12-16T07:28:55.118 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: Added host 'smithi007' with addr '172.21.15.7' 2023-12-16T07:28:56.010 INFO:teuthology.orchestra.run.smithi007.stderr:Deploying unmanaged mon service... 2023-12-16T07:28:56.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:56 smithi007 bash[25009]: audit 2023-12-16T07:28:55.115118+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' 2023-12-16T07:28:56.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:56 smithi007 bash[25009]: cephadm 2023-12-16T07:28:55.115799+0000 mgr.y (mgr.14120) 13 : cephadm [INF] Added host smithi007 2023-12-16T07:28:56.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:56 smithi007 bash[25009]: audit 2023-12-16T07:28:55.122664+0000 mon.a (mon.0) 58 : audit [DBG] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:28:57.981 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: Scheduled mon update... 2023-12-16T07:28:58.934 INFO:teuthology.orchestra.run.smithi007.stderr:Deploying unmanaged mgr service... 2023-12-16T07:28:59.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:58 smithi007 bash[25009]: audit 2023-12-16T07:28:57.972431+0000 mgr.y (mgr.14120) 14 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:28:59.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:58 smithi007 bash[25009]: cephadm 2023-12-16T07:28:57.975154+0000 mgr.y (mgr.14120) 15 : cephadm [INF] Saving service mon spec with placement count:5 2023-12-16T07:28:59.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:58 smithi007 bash[25009]: audit 2023-12-16T07:28:57.979722+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' 2023-12-16T07:28:59.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:58 smithi007 bash[25009]: audit 2023-12-16T07:28:58.578431+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' 2023-12-16T07:28:59.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:28:58 smithi007 bash[25009]: audit 2023-12-16T07:28:58.864264+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' 2023-12-16T07:29:00.529 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-12-16T07:29:01.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:01 smithi007 bash[25009]: audit 2023-12-16T07:29:00.523484+0000 mgr.y (mgr.14120) 16 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:29:01.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:01 smithi007 bash[25009]: cephadm 2023-12-16T07:29:00.524616+0000 mgr.y (mgr.14120) 17 : cephadm [INF] Saving service mgr spec with placement count:2 2023-12-16T07:29:01.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:01 smithi007 bash[25009]: audit 2023-12-16T07:29:00.528067+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' 2023-12-16T07:29:03.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:02 smithi007 bash[25009]: audit 2023-12-16T07:29:01.998568+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' 2023-12-16T07:29:03.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:02 smithi007 bash[25009]: audit 2023-12-16T07:29:02.460082+0000 mon.a (mon.0) 64 : audit [INF] from='client.? 172.21.15.7:0/1978603198' entity='client.admin' 2023-12-16T07:29:03.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:02 smithi007 bash[25009]: audit 2023-12-16T07:29:02.498896+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.14120 172.21.15.7:0/970784765' entity='mgr.y' 2023-12-16T07:29:04.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:04 smithi007 bash[25009]: cluster 2023-12-16T07:29:03.022124+0000 mgr.y (mgr.14120) 18 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:05.290 INFO:teuthology.orchestra.run.smithi007.stderr:Enabling the dashboard module... 2023-12-16T07:29:06.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:05 smithi007 bash[25009]: audit 2023-12-16T07:29:04.675709+0000 mon.a (mon.0) 66 : audit [INF] from='client.? 172.21.15.7:0/3285211286' entity='client.admin' 2023-12-16T07:29:06.804 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:06 smithi007 bash[25009]: cluster 2023-12-16T07:29:05.022550+0000 mgr.y (mgr.14120) 19 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:08.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:07 smithi007 bash[25009]: cluster 2023-12-16T07:29:07.022983+0000 mgr.y (mgr.14120) 20 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:08.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:07 smithi007 bash[25009]: audit 2023-12-16T07:29:07.237778+0000 mon.a (mon.0) 67 : audit [INF] from='client.? 172.21.15.7:0/3044527262' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-12-16T07:29:08.316 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:08 smithi007 bash[25282]: ignoring --setuser ceph since I am not root 2023-12-16T07:29:08.316 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:08 smithi007 bash[25282]: ignoring --setgroup ceph since I am not root 2023-12-16T07:29:08.316 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:08 smithi007 bash[25282]: debug 2023-12-16T07:29:08.178+0000 7fba79830000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T07:29:08.965 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:08 smithi007 bash[25282]: debug 2023-12-16T07:29:08.874+0000 7fba79830000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T07:29:09.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:08 smithi007 bash[25009]: audit 2023-12-16T07:29:07.969470+0000 mon.a (mon.0) 68 : audit [INF] from='client.? 172.21.15.7:0/3044527262' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-12-16T07:29:09.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:08 smithi007 bash[25009]: cluster 2023-12-16T07:29:07.969568+0000 mon.a (mon.0) 69 : cluster [DBG] mgrmap e9: y(active, since 25s) 2023-12-16T07:29:09.316 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:09 smithi007 bash[25282]: debug 2023-12-16T07:29:09.002+0000 7fba79830000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T07:29:09.316 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:09 smithi007 bash[25282]: debug 2023-12-16T07:29:09.170+0000 7fba79830000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:29:09.620 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:09 smithi007 bash[25282]: debug 2023-12-16T07:29:09.522+0000 7fba79830000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T07:29:10.014 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2023-12-16T07:29:10.014 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 9, 2023-12-16T07:29:10.014 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": true, 2023-12-16T07:29:10.014 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "active_name": "y", 2023-12-16T07:29:10.014 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-16T07:29:10.014 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2023-12-16T07:29:10.040 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:09 smithi007 bash[25282]: debug 2023-12-16T07:29:09.618+0000 7fba79830000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T07:29:10.040 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:09 smithi007 bash[25282]: debug 2023-12-16T07:29:09.826+0000 7fba79830000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T07:29:10.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:10 smithi007 bash[25009]: audit 2023-12-16T07:29:10.011011+0000 mon.a (mon.0) 70 : audit [DBG] from='client.? 172.21.15.7:0/3300761970' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-16T07:29:10.671 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for the mgr to restart... 2023-12-16T07:29:10.671 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mgr epoch 9... 2023-12-16T07:29:11.021 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:10 smithi007 bash[25282]: debug 2023-12-16T07:29:10.693+0000 7fba79830000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T07:29:11.022 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:10 smithi007 bash[25282]: debug 2023-12-16T07:29:10.805+0000 7fba79830000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T07:29:11.315 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:11 smithi007 bash[25282]: debug 2023-12-16T07:29:11.013+0000 7fba79830000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T07:29:11.722 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:11 smithi007 bash[25282]: debug 2023-12-16T07:29:11.377+0000 7fba79830000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T07:29:12.065 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:11 smithi007 bash[25282]: debug 2023-12-16T07:29:11.717+0000 7fba79830000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T07:29:12.391 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:12 smithi007 bash[25282]: debug 2023-12-16T07:29:12.105+0000 7fba79830000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T07:29:12.815 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:12 smithi007 bash[25282]: debug 2023-12-16T07:29:12.385+0000 7fba79830000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:29:13.365 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:13 smithi007 bash[25282]: debug 2023-12-16T07:29:13.089+0000 7fba79830000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T07:29:13.366 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:13 smithi007 bash[25282]: debug 2023-12-16T07:29:13.197+0000 7fba79830000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T07:29:13.620 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:13 smithi007 bash[25282]: debug 2023-12-16T07:29:13.361+0000 7fba79830000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T07:29:13.620 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:13 smithi007 bash[25282]: debug 2023-12-16T07:29:13.473+0000 7fba79830000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T07:29:13.915 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:13 smithi007 bash[25282]: debug 2023-12-16T07:29:13.617+0000 7fba79830000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T07:29:14.226 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:13 smithi007 bash[25282]: debug 2023-12-16T07:29:13.913+0000 7fba79830000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T07:29:14.226 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:14 smithi007 bash[25282]: debug 2023-12-16T07:29:14.033+0000 7fba79830000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T07:29:14.565 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:14 smithi007 bash[25282]: debug 2023-12-16T07:29:14.221+0000 7fba79830000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T07:29:14.565 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:14 smithi007 bash[25282]: debug 2023-12-16T07:29:14.345+0000 7fba79830000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T07:29:15.565 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:15 smithi007 bash[25282]: debug 2023-12-16T07:29:15.225+0000 7fba79830000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T07:29:16.330 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2023-12-16T07:29:16.331 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2023-12-16T07:29:16.331 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "initialized": true 2023-12-16T07:29:16.331 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2023-12-16T07:29:16.532 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25009]: cluster 2023-12-16T07:29:15.253097+0000 mon.a (mon.0) 71 : cluster [INF] Active manager daemon y restarted 2023-12-16T07:29:16.532 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25009]: cluster 2023-12-16T07:29:15.254052+0000 mon.a (mon.0) 72 : cluster [INF] Activating manager daemon y 2023-12-16T07:29:16.532 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25009]: cluster 2023-12-16T07:29:15.257550+0000 mon.a (mon.0) 73 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-12-16T07:29:16.532 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25009]: cluster 2023-12-16T07:29:15.310226+0000 mon.a (mon.0) 74 : cluster [DBG] mgrmap e10: y(active, starting, since 0.056291s) 2023-12-16T07:29:16.533 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25009]: audit 2023-12-16T07:29:15.312182+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:29:16.533 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25009]: audit 2023-12-16T07:29:15.314233+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:29:16.533 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25009]: audit 2023-12-16T07:29:15.315877+0000 mon.a (mon.0) 77 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:29:16.533 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25009]: audit 2023-12-16T07:29:15.316268+0000 mon.a (mon.0) 78 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:29:16.533 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25009]: audit 2023-12-16T07:29:15.316514+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:29:16.533 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25009]: cluster 2023-12-16T07:29:15.326306+0000 mon.a (mon.0) 80 : cluster [INF] Manager daemon y is now available 2023-12-16T07:29:16.533 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25009]: audit 2023-12-16T07:29:15.355232+0000 mon.a (mon.0) 81 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:29:16.533 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25009]: audit 2023-12-16T07:29:15.357667+0000 mon.a (mon.0) 82 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:29:16.533 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25009]: audit 2023-12-16T07:29:15.382893+0000 mon.a (mon.0) 83 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:29:16.533 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25009]: audit 2023-12-16T07:29:15.385545+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:29:16.815 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25282]: [16/Dec/2023:07:29:16] ENGINE Bus STARTING 2023-12-16T07:29:16.816 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25282]: [16/Dec/2023:07:29:16] ENGINE Serving on https://172.21.15.7:7150 2023-12-16T07:29:16.816 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:29:16 smithi007 bash[25282]: [16/Dec/2023:07:29:16] ENGINE Bus STARTED 2023-12-16T07:29:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:mgr epoch 9 is available 2023-12-16T07:29:17.030 INFO:teuthology.orchestra.run.smithi007.stderr:Generating a dashboard self-signed certificate... 2023-12-16T07:29:17.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:17 smithi007 bash[25009]: cluster 2023-12-16T07:29:16.315292+0000 mon.a (mon.0) 85 : cluster [DBG] mgrmap e11: y(active, since 1.06136s) 2023-12-16T07:29:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:17 smithi007 bash[25009]: cephadm 2023-12-16T07:29:16.599603+0000 mgr.y (mgr.14152) 3 : cephadm [INF] [16/Dec/2023:07:29:16] ENGINE Bus STARTING 2023-12-16T07:29:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:17 smithi007 bash[25009]: cephadm 2023-12-16T07:29:16.731625+0000 mgr.y (mgr.14152) 4 : cephadm [INF] [16/Dec/2023:07:29:16] ENGINE Serving on https://172.21.15.7:7150 2023-12-16T07:29:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:17 smithi007 bash[25009]: cephadm 2023-12-16T07:29:16.732353+0000 mgr.y (mgr.14152) 5 : cephadm [INF] [16/Dec/2023:07:29:16] ENGINE Bus STARTED 2023-12-16T07:29:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:17 smithi007 bash[25009]: audit 2023-12-16T07:29:16.738708+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:18.800 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: Self-signed certificate created 2023-12-16T07:29:19.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:18 smithi007 bash[25009]: cluster 2023-12-16T07:29:17.965752+0000 mon.a (mon.0) 87 : cluster [DBG] mgrmap e12: y(active, since 2s) 2023-12-16T07:29:19.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:18 smithi007 bash[25009]: audit 2023-12-16T07:29:18.796233+0000 mon.a (mon.0) 88 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:19.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:18 smithi007 bash[25009]: audit 2023-12-16T07:29:18.798653+0000 mon.a (mon.0) 89 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:19.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:18 smithi007 bash[25009]: audit 2023-12-16T07:29:18.836045+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:19.548 INFO:teuthology.orchestra.run.smithi007.stderr:Creating initial admin user... 2023-12-16T07:29:20.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:19 smithi007 bash[25009]: audit 2023-12-16T07:29:18.482410+0000 mgr.y (mgr.14152) 6 : audit [DBG] from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:29:20.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:19 smithi007 bash[25009]: audit 2023-12-16T07:29:19.296258+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:21.256 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$CTfmCKnUMJ2nlo7thfQDpegrusX4XrKd8UuMxQhZg/1c64ZlmSlpa", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1702711761, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-12-16T07:29:21.431 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:21 smithi007 bash[25009]: audit 2023-12-16T07:29:21.005717+0000 mgr.y (mgr.14152) 7 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:29:21.985 INFO:teuthology.orchestra.run.smithi007.stderr:Fetching dashboard port number... 2023-12-16T07:29:22.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:22 smithi007 bash[25009]: audit 2023-12-16T07:29:21.254326+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:23.562 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:23 smithi007 bash[25009]: cluster 2023-12-16T07:29:22.262123+0000 mon.a (mon.0) 93 : cluster [DBG] mgrmap e13: y(active, since 7s) 2023-12-16T07:29:23.874 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 8443 2023-12-16T07:29:24.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:24 smithi007 bash[25009]: audit 2023-12-16T07:29:23.872907+0000 mon.a (mon.0) 94 : audit [DBG] from='client.? 172.21.15.7:0/4132641080' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-12-16T07:29:25.002 INFO:teuthology.orchestra.run.smithi007.stderr:firewalld does not appear to be present 2023-12-16T07:29:25.002 INFO:teuthology.orchestra.run.smithi007.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-12-16T07:29:25.006 INFO:teuthology.orchestra.run.smithi007.stderr:Ceph Dashboard is now available at: 2023-12-16T07:29:25.006 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-12-16T07:29:25.007 INFO:teuthology.orchestra.run.smithi007.stderr: URL: https://smithi007.front.sepia.ceph.com:8443/ 2023-12-16T07:29:25.007 INFO:teuthology.orchestra.run.smithi007.stderr: User: admin 2023-12-16T07:29:25.007 INFO:teuthology.orchestra.run.smithi007.stderr: Password: 7zv68fx3m1 2023-12-16T07:29:25.007 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-12-16T07:29:25.007 INFO:teuthology.orchestra.run.smithi007.stderr:Enabling autotune for osd_memory_target 2023-12-16T07:29:28.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2023-12-16T07:29:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:You can access the Ceph CLI with: 2023-12-16T07:29:29.141 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-12-16T07:29:29.141 INFO:teuthology.orchestra.run.smithi007.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-12-16T07:29:29.141 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-12-16T07:29:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:Please consider enabling telemetry to help improve Ceph: 2023-12-16T07:29:29.141 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-12-16T07:29:29.141 INFO:teuthology.orchestra.run.smithi007.stderr: ceph telemetry on 2023-12-16T07:29:29.141 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-12-16T07:29:29.141 INFO:teuthology.orchestra.run.smithi007.stderr:For more information see: 2023-12-16T07:29:29.141 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-12-16T07:29:29.141 INFO:teuthology.orchestra.run.smithi007.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2023-12-16T07:29:29.141 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-12-16T07:29:29.142 INFO:teuthology.orchestra.run.smithi007.stderr:Bootstrap complete. 2023-12-16T07:29:29.178 INFO:tasks.cephadm:Fetching config... 2023-12-16T07:29:29.178 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-12-16T07:29:29.179 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-12-16T07:29:29.185 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-12-16T07:29:29.185 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-12-16T07:29:29.186 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-12-16T07:29:29.191 INFO:tasks.cephadm:Fetching mon keyring... 2023-12-16T07:29:29.192 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-12-16T07:29:29.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/keyring of=/dev/stdout 2023-12-16T07:29:29.248 INFO:tasks.cephadm:Fetching pub ssh key... 2023-12-16T07:29:29.248 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-12-16T07:29:29.249 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-12-16T07:29:29.253 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-12-16T07:29:29.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0in26ZtYrIbvbKvwXM8aK4lBMIBt2uayQJnDbfmcckmnd6d5HLtHOSCypMABi69gm8xZs/yaMGKXFnBx29Dl085B7IgzGWBnUY6p6PZhfh56QC8K6ZJcsrX+ov1Km3wiMg19uUjT1y0AWXEemUz3jBGUHc88EZfsxj71RyF8Svd7aQuCg0u48um9XxqmmsiQ1z82C6AkMoND99ucriAvV7/BhV6WC2CQ6a3uf5ap/EZkc1zhgv+26csHzVAQObGTmC0p3KeuUcnSXXARnhC38kq3RAzgkas+yeUdpI3D3OfvJjd4FDzABSUxbVIo/+BNOd5XblRaf7ivKyboOZ/36kX9rdDhy1wSAE1VlA/Qo21vLEV5j403jJ3rZv75kGkiHgGGitdR8ZQ9dOhbg1ps1dciC0AI1rHLRgf8fQJ7MaleNerJ/zCpnnCCxQcNeK+T+UoQag6nI3BUNCTMGIs+GNTxtaX+h5NtisssyD2VWZIAT/Gay8Rati56Ay59o87s= ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-16T07:29:29.278 INFO:teuthology.orchestra.run.smithi007.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0in26ZtYrIbvbKvwXM8aK4lBMIBt2uayQJnDbfmcckmnd6d5HLtHOSCypMABi69gm8xZs/yaMGKXFnBx29Dl085B7IgzGWBnUY6p6PZhfh56QC8K6ZJcsrX+ov1Km3wiMg19uUjT1y0AWXEemUz3jBGUHc88EZfsxj71RyF8Svd7aQuCg0u48um9XxqmmsiQ1z82C6AkMoND99ucriAvV7/BhV6WC2CQ6a3uf5ap/EZkc1zhgv+26csHzVAQObGTmC0p3KeuUcnSXXARnhC38kq3RAzgkas+yeUdpI3D3OfvJjd4FDzABSUxbVIo/+BNOd5XblRaf7ivKyboOZ/36kX9rdDhy1wSAE1VlA/Qo21vLEV5j403jJ3rZv75kGkiHgGGitdR8ZQ9dOhbg1ps1dciC0AI1rHLRgf8fQJ7MaleNerJ/zCpnnCCxQcNeK+T+UoQag6nI3BUNCTMGIs+GNTxtaX+h5NtisssyD2VWZIAT/Gay8Rati56Ay59o87s= ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:29:29.289 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0in26ZtYrIbvbKvwXM8aK4lBMIBt2uayQJnDbfmcckmnd6d5HLtHOSCypMABi69gm8xZs/yaMGKXFnBx29Dl085B7IgzGWBnUY6p6PZhfh56QC8K6ZJcsrX+ov1Km3wiMg19uUjT1y0AWXEemUz3jBGUHc88EZfsxj71RyF8Svd7aQuCg0u48um9XxqmmsiQ1z82C6AkMoND99ucriAvV7/BhV6WC2CQ6a3uf5ap/EZkc1zhgv+26csHzVAQObGTmC0p3KeuUcnSXXARnhC38kq3RAzgkas+yeUdpI3D3OfvJjd4FDzABSUxbVIo/+BNOd5XblRaf7ivKyboOZ/36kX9rdDhy1wSAE1VlA/Qo21vLEV5j403jJ3rZv75kGkiHgGGitdR8ZQ9dOhbg1ps1dciC0AI1rHLRgf8fQJ7MaleNerJ/zCpnnCCxQcNeK+T+UoQag6nI3BUNCTMGIs+GNTxtaX+h5NtisssyD2VWZIAT/Gay8Rati56Ay59o87s= ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-16T07:29:29.313 INFO:teuthology.orchestra.run.smithi169.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0in26ZtYrIbvbKvwXM8aK4lBMIBt2uayQJnDbfmcckmnd6d5HLtHOSCypMABi69gm8xZs/yaMGKXFnBx29Dl085B7IgzGWBnUY6p6PZhfh56QC8K6ZJcsrX+ov1Km3wiMg19uUjT1y0AWXEemUz3jBGUHc88EZfsxj71RyF8Svd7aQuCg0u48um9XxqmmsiQ1z82C6AkMoND99ucriAvV7/BhV6WC2CQ6a3uf5ap/EZkc1zhgv+26csHzVAQObGTmC0p3KeuUcnSXXARnhC38kq3RAzgkas+yeUdpI3D3OfvJjd4FDzABSUxbVIo/+BNOd5XblRaf7ivKyboOZ/36kX9rdDhy1wSAE1VlA/Qo21vLEV5j403jJ3rZv75kGkiHgGGitdR8ZQ9dOhbg1ps1dciC0AI1rHLRgf8fQJ7MaleNerJ/zCpnnCCxQcNeK+T+UoQag6nI3BUNCTMGIs+GNTxtaX+h5NtisssyD2VWZIAT/Gay8Rati56Ay59o87s= ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:29:29.324 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-12-16T07:29:29.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:29 smithi007 bash[25009]: audit 2023-12-16T07:29:28.499302+0000 mon.a (mon.0) 95 : audit [INF] from='client.? 172.21.15.7:0/195366548' entity='client.admin' 2023-12-16T07:29:34.122 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-12-16T07:29:34.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-12-16T07:29:34.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:34 smithi007 bash[25009]: audit 2023-12-16T07:29:33.139997+0000 mon.a (mon.0) 96 : audit [INF] from='client.? 172.21.15.7:0/2626898066' entity='client.admin' 2023-12-16T07:29:36.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:36 smithi007 bash[25009]: cluster 2023-12-16T07:29:35.317268+0000 mgr.y (mgr.14152) 8 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:37.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:37 smithi007 bash[25009]: audit 2023-12-16T07:29:36.824420+0000 mon.a (mon.0) 97 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:37.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:37 smithi007 bash[25009]: audit 2023-12-16T07:29:36.825823+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:29:37.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:37 smithi007 bash[25009]: audit 2023-12-16T07:29:36.832331+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:37.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:37 smithi007 bash[25009]: audit 2023-12-16T07:29:36.834290+0000 mon.a (mon.0) 100 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:29:37.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:37 smithi007 bash[25009]: audit 2023-12-16T07:29:36.841821+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:37.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:37 smithi007 bash[25009]: audit 2023-12-16T07:29:37.659530+0000 mon.a (mon.0) 102 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:37.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:37 smithi007 bash[25009]: audit 2023-12-16T07:29:37.709913+0000 mon.a (mon.0) 103 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:29:37.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:37 smithi007 bash[25009]: audit 2023-12-16T07:29:37.712663+0000 mon.a (mon.0) 104 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:29:37.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:37 smithi007 bash[25009]: audit 2023-12-16T07:29:37.714236+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:29:38.577 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi169 2023-12-16T07:29:38.577 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T07:29:38.577 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/etc/ceph/ceph.conf 2023-12-16T07:29:38.584 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T07:29:38.584 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:29:38.635 INFO:tasks.cephadm:Adding host smithi169 to orchestrator... 2023-12-16T07:29:38.635 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch host add smithi169 2023-12-16T07:29:39.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:38 smithi007 bash[25009]: cluster 2023-12-16T07:29:37.317708+0000 mgr.y (mgr.14152) 9 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:39.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:38 smithi007 bash[25009]: audit 2023-12-16T07:29:37.654344+0000 mgr.y (mgr.14152) 10 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:29:39.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:38 smithi007 bash[25009]: cephadm 2023-12-16T07:29:37.716532+0000 mgr.y (mgr.14152) 11 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:29:39.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:38 smithi007 bash[25009]: cephadm 2023-12-16T07:29:37.840933+0000 mgr.y (mgr.14152) 12 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:29:39.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:38 smithi007 bash[25009]: audit 2023-12-16T07:29:38.091844+0000 mon.a (mon.0) 106 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:39.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:38 smithi007 bash[25009]: audit 2023-12-16T07:29:38.097649+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:41.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:40 smithi007 bash[25009]: cluster 2023-12-16T07:29:39.318227+0000 mgr.y (mgr.14152) 13 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:42.674 INFO:teuthology.orchestra.run.smithi007.stdout:Added host 'smithi169' with addr '172.21.15.169' 2023-12-16T07:29:43.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:42 smithi007 bash[25009]: cluster 2023-12-16T07:29:41.318784+0000 mgr.y (mgr.14152) 14 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:43.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:42 smithi007 bash[25009]: audit 2023-12-16T07:29:41.608113+0000 mgr.y (mgr.14152) 15 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi169", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:29:43.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:42 smithi007 bash[25009]: audit 2023-12-16T07:29:42.672232+0000 mon.a (mon.0) 108 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:43.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:42 smithi007 bash[25009]: audit 2023-12-16T07:29:42.691120+0000 mon.a (mon.0) 109 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:29:43.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:42 smithi007 bash[25009]: audit 2023-12-16T07:29:42.693853+0000 mon.a (mon.0) 110 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:29:43.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:42 smithi007 bash[25009]: audit 2023-12-16T07:29:42.695628+0000 mon.a (mon.0) 111 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:29:43.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:42 smithi007 bash[25009]: audit 2023-12-16T07:29:42.970409+0000 mon.a (mon.0) 112 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:43.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch host ls --format=json 2023-12-16T07:29:44.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:43 smithi007 bash[25009]: cephadm 2023-12-16T07:29:42.029705+0000 mgr.y (mgr.14152) 16 : cephadm [INF] Deploying cephadm binary to smithi169 2023-12-16T07:29:44.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:43 smithi007 bash[25009]: cephadm 2023-12-16T07:29:42.673113+0000 mgr.y (mgr.14152) 17 : cephadm [INF] Added host smithi169 2023-12-16T07:29:44.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:43 smithi007 bash[25009]: audit 2023-12-16T07:29:43.721066+0000 mon.a (mon.0) 113 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:45.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:44 smithi007 bash[25009]: cluster 2023-12-16T07:29:43.319223+0000 mgr.y (mgr.14152) 18 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:45.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:44 smithi007 bash[25009]: audit 2023-12-16T07:29:44.248901+0000 mon.a (mon.0) 114 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:46.505 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-16T07:29:46.505 INFO:teuthology.orchestra.run.smithi007.stdout:[{"addr": "172.21.15.7", "hostname": "smithi007", "labels": [], "status": ""}, {"addr": "172.21.15.169", "hostname": "smithi169", "labels": [], "status": ""}] 2023-12-16T07:29:47.215 INFO:tasks.cephadm:Setting crush tunables to default 2023-12-16T07:29:47.216 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph osd crush tunables default 2023-12-16T07:29:47.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:46 smithi007 bash[25009]: cluster 2023-12-16T07:29:45.319642+0000 mgr.y (mgr.14152) 19 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:48.265 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:47 smithi007 bash[25009]: audit 2023-12-16T07:29:46.503886+0000 mgr.y (mgr.14152) 20 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T07:29:49.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:48 smithi007 bash[25009]: cluster 2023-12-16T07:29:47.320079+0000 mgr.y (mgr.14152) 21 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:50.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:50 smithi007 bash[25009]: audit 2023-12-16T07:29:49.172682+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:50.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:50 smithi007 bash[25009]: audit 2023-12-16T07:29:49.179126+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:50.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:50 smithi007 bash[25009]: audit 2023-12-16T07:29:49.180630+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:29:50.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:50 smithi007 bash[25009]: audit 2023-12-16T07:29:49.186529+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:50.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:50 smithi007 bash[25009]: cluster 2023-12-16T07:29:49.320512+0000 mgr.y (mgr.14152) 22 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:51.201 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2023-12-16T07:29:51.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:51 smithi007 bash[25009]: audit 2023-12-16T07:29:50.369279+0000 mon.a (mon.0) 119 : audit [INF] from='client.? 172.21.15.7:0/2702297157' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-16T07:29:51.956 INFO:tasks.cephadm:Adding mon.a on smithi007 2023-12-16T07:29:51.956 INFO:tasks.cephadm:Adding mon.c on smithi007 2023-12-16T07:29:51.956 INFO:tasks.cephadm:Adding mon.b on smithi169 2023-12-16T07:29:51.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch apply mon '3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi169:172.21.15.169=b' 2023-12-16T07:29:52.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:52 smithi007 bash[25009]: audit 2023-12-16T07:29:51.194448+0000 mon.a (mon.0) 120 : audit [INF] from='client.? 172.21.15.7:0/2702297157' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-16T07:29:52.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:52 smithi007 bash[25009]: cluster 2023-12-16T07:29:51.194584+0000 mon.a (mon.0) 121 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-16T07:29:52.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:52 smithi007 bash[25009]: cluster 2023-12-16T07:29:51.320931+0000 mgr.y (mgr.14152) 23 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:54.780 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled mon update... 2023-12-16T07:29:54.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:54 smithi007 bash[25009]: cluster 2023-12-16T07:29:53.321430+0000 mgr.y (mgr.14152) 24 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:55.453 DEBUG:teuthology.orchestra.run.smithi007:mon.c> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.c.service 2023-12-16T07:29:55.455 DEBUG:teuthology.orchestra.run.smithi169:mon.b> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.b.service 2023-12-16T07:29:55.458 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-12-16T07:29:55.458 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T07:29:55.962 INFO:journalctl@ceph.mon.b.smithi169.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:29:56.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:55 smithi007 bash[25009]: audit 2023-12-16T07:29:54.767707+0000 mgr.y (mgr.14152) 25 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi169:172.21.15.169=b", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:29:56.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:55 smithi007 bash[25009]: cephadm 2023-12-16T07:29:54.772975+0000 mgr.y (mgr.14152) 26 : cephadm [INF] Saving service mon spec with placement smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi169:172.21.15.169=b;count:3 2023-12-16T07:29:56.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:55 smithi007 bash[25009]: audit 2023-12-16T07:29:54.778493+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:56.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:55 smithi007 bash[25009]: audit 2023-12-16T07:29:54.817011+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:29:56.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:55 smithi007 bash[25009]: audit 2023-12-16T07:29:54.821740+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:29:56.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:55 smithi007 bash[25009]: audit 2023-12-16T07:29:54.823680+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:29:56.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:55 smithi007 bash[25009]: cephadm 2023-12-16T07:29:54.826380+0000 mgr.y (mgr.14152) 27 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:29:56.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:55 smithi007 bash[25009]: cephadm 2023-12-16T07:29:54.940106+0000 mgr.y (mgr.14152) 28 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:29:56.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:55 smithi007 bash[25009]: audit 2023-12-16T07:29:55.164388+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:56.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:55 smithi007 bash[25009]: audit 2023-12-16T07:29:55.170721+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:29:56.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:55 smithi007 bash[25009]: audit 2023-12-16T07:29:55.174360+0000 mon.a (mon.0) 128 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:29:56.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:55 smithi007 bash[25009]: audit 2023-12-16T07:29:55.176049+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:29:56.681 INFO:journalctl@ceph.mon.c.smithi007.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:29:57.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:56 smithi007 bash[25009]: cephadm 2023-12-16T07:29:55.177393+0000 mgr.y (mgr.14152) 29 : cephadm [INF] Deploying daemon mon.c on smithi007 2023-12-16T07:29:57.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:56 smithi007 bash[25009]: cluster 2023-12-16T07:29:55.322003+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:58.330 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T07:29:58.330 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":1,"fsid":"8b71d3dc-9be4-11ee-95a5-87774f69a715","modified":"2023-12-16T07:28:06.053524Z","created":"2023-12-16T07:28:06.053524Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-16T07:29:58.333 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 1 2023-12-16T07:29:59.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:58 smithi007 bash[25009]: cluster 2023-12-16T07:29:57.322491+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:29:59.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:29:58 smithi007 bash[25009]: audit 2023-12-16T07:29:58.328677+0000 mon.a (mon.0) 130 : audit [DBG] from='client.? 172.21.15.169:0/2811537117' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T07:29:59.626 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:29:59 smithi007 systemd[1]: Started Ceph mon.c for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:29:59.996 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-12-16T07:29:59.996 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-16T07:30:00.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[25009]: cluster 2023-12-16T07:29:59.322809+0000 mgr.y (mgr.14152) 32 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:00.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[25009]: audit 2023-12-16T07:29:59.632463+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:00.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[25009]: audit 2023-12-16T07:29:59.633755+0000 mon.a (mon.0) 132 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:30:00.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[25009]: audit 2023-12-16T07:29:59.634670+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:00.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[25009]: cephadm 2023-12-16T07:29:59.635673+0000 mgr.y (mgr.14152) 33 : cephadm [INF] Deploying daemon mon.b on smithi169 2023-12-16T07:30:00.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.656+0000 7fe9e53df880 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-16T07:30:00.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.656+0000 7fe9e53df880 0 ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 6 2023-12-16T07:30:00.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.656+0000 7fe9e53df880 0 pidfile_write: ignore empty --pid-file 2023-12-16T07:30:00.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 0 load: jerasure load: lrc 2023-12-16T07:30:00.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: RocksDB version: 6.15.5 2023-12-16T07:30:00.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-16T07:30:00.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Compile date Apr 18 2022 2023-12-16T07:30:00.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: DB SUMMARY 2023-12-16T07:30:00.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: DB Session ID: XROYWWR2CCF9UGV729AJ 2023-12-16T07:30:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: CURRENT file: CURRENT 2023-12-16T07:30:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: IDENTITY file: IDENTITY 2023-12-16T07:30:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2023-12-16T07:30:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-12-16T07:30:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-12-16T07:30:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.error_if_exists: 0 2023-12-16T07:30:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.create_if_missing: 0 2023-12-16T07:30:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.paranoid_checks: 1 2023-12-16T07:30:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-12-16T07:30:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.env: 0x555a25f0e860 2023-12-16T07:30:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.fs: Posix File System 2023-12-16T07:30:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.info_log: 0x555a26d5be00 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.statistics: (nil) 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.use_fsync: 0 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_log_file_size: 0 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.allow_fallocate: 1 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.use_direct_reads: 0 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.create_missing_column_families: 0 2023-12-16T07:30:00.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.db_log_dir: 2023-12-16T07:30:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2023-12-16T07:30:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-16T07:30:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-16T07:30:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-16T07:30:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-16T07:30:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-16T07:30:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-16T07:30:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.advise_random_on_open: 1 2023-12-16T07:30:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-16T07:30:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.write_buffer_manager: 0x555a26e4c300 2023-12-16T07:30:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-16T07:30:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-16T07:30:00.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-16T07:30:00.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-16T07:30:00.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.rate_limiter: (nil) 2023-12-16T07:30:00.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-16T07:30:00.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-16T07:30:00.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-16T07:30:00.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-16T07:30:00.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.unordered_write: 0 2023-12-16T07:30:00.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-16T07:30:00.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-16T07:30:00.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-16T07:30:00.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-16T07:30:00.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.row_cache: None 2023-12-16T07:30:00.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.wal_filter: None 2023-12-16T07:30:00.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-16T07:30:00.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-16T07:30:00.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.preserve_deletes: 0 2023-12-16T07:30:00.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.two_write_queues: 0 2023-12-16T07:30:00.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.manual_wal_flush: 0 2023-12-16T07:30:00.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.atomic_flush: 0 2023-12-16T07:30:00.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-16T07:30:00.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-16T07:30:00.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-16T07:30:00.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.log_readahead_size: 0 2023-12-16T07:30:00.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-12-16T07:30:00.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.best_efforts_recovery: 0 2023-12-16T07:30:00.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-12-16T07:30:00.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-12-16T07:30:00.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.allow_data_in_errors: 0 2023-12-16T07:30:00.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.db_host_id: __hostname__ 2023-12-16T07:30:00.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_background_jobs: 2 2023-12-16T07:30:00.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_background_compactions: -1 2023-12-16T07:30:00.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_subcompactions: 1 2023-12-16T07:30:00.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-16T07:30:00.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-16T07:30:00.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-16T07:30:00.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_total_wal_size: 0 2023-12-16T07:30:00.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-16T07:30:00.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-16T07:30:00.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-16T07:30:00.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-16T07:30:00.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_open_files: -1 2023-12-16T07:30:00.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.bytes_per_sync: 0 2023-12-16T07:30:00.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-16T07:30:00.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-16T07:30:00.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-16T07:30:00.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_background_flushes: -1 2023-12-16T07:30:00.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Compression algorithms supported: 2023-12-16T07:30:00.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-16T07:30:00.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: kZSTD supported: 0 2023-12-16T07:30:00.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: kXpressCompression supported: 0 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: kLZ4Compression supported: 1 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: kBZip2Compression supported: 0 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: kZlibCompression supported: 1 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: kSnappyCompression supported: 1 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.merge_operator: 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compaction_filter: None 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compaction_filter_factory: None 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.sst_partitioner_factory: None 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-16T07:30:00.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-16T07:30:00.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555a26d29d00) 2023-12-16T07:30:00.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cache_index_and_filter_blocks: 1 2023-12-16T07:30:00.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-16T07:30:00.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-16T07:30:00.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: pin_top_level_index_and_filter: 1 2023-12-16T07:30:00.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: index_type: 0 2023-12-16T07:30:00.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: data_block_index_type: 0 2023-12-16T07:30:00.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: index_shortening: 1 2023-12-16T07:30:00.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: data_block_hash_table_util_ratio: 0.750000 2023-12-16T07:30:00.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: hash_index_allow_collision: 1 2023-12-16T07:30:00.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: checksum: 1 2023-12-16T07:30:00.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: no_block_cache: 0 2023-12-16T07:30:00.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: block_cache: 0x555a26d90170 2023-12-16T07:30:00.827 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: block_cache_name: BinnedLRUCache 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: block_cache_options: 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: capacity : 536870912 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: num_shard_bits : 4 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: strict_capacity_limit : 0 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: high_pri_pool_ratio: 0.000 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: block_cache_compressed: (nil) 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: persistent_cache: (nil) 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: block_size: 4096 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: block_size_deviation: 10 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: block_restart_interval: 16 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: index_block_restart_interval: 1 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: metadata_block_size: 4096 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: partition_filters: 0 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: use_delta_encoding: 1 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-16T07:30:00.828 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: whole_key_filtering: 1 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: verify_compression: 0 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: read_amp_bytes_per_bit: 0 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: format_version: 4 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: enable_index_compression: 1 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: block_align: 0 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compression: NoCompression 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.num_levels: 7 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-16T07:30:00.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-16T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-16T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-16T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-16T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-16T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-16T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-12-16T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-16T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-16T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compression_opts.level: 32767 2023-12-16T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-16T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-16T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-16T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-12-16T07:30:00.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compression_opts.enabled: false 2023-12-16T07:30:00.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-16T07:30:00.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-16T07:30:00.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-16T07:30:00.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-16T07:30:00.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-16T07:30:00.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-16T07:30:00.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-16T07:30:00.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-16T07:30:00.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-16T07:30:00.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-16T07:30:00.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-16T07:30:00.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-16T07:30:00.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-16T07:30:00.831 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-16T07:30:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-16T07:30:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-16T07:30:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-16T07:30:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.arena_block_size: 4194304 2023-12-16T07:30:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-16T07:30:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-16T07:30:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-16T07:30:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-16T07:30:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-16T07:30:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-16T07:30:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-16T07:30:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-16T07:30:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-16T07:30:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-16T07:30:00.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-16T07:30:00.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-16T07:30:00.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-16T07:30:00.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-16T07:30:00.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.table_properties_collectors: 2023-12-16T07:30:00.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.inplace_update_support: 0 2023-12-16T07:30:00.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-16T07:30:00.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-16T07:30:00.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-16T07:30:00.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-16T07:30:00.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.bloom_locality: 0 2023-12-16T07:30:00.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.max_successive_merges: 0 2023-12-16T07:30:00.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-16T07:30:00.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-16T07:30:00.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.force_consistency_checks: 1 2023-12-16T07:30:00.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-16T07:30:00.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.ttl: 2592000 2023-12-16T07:30:00.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-16T07:30:00.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.enable_blob_files: false 2023-12-16T07:30:00.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.min_blob_size: 0 2023-12-16T07:30:00.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.blob_file_size: 268435456 2023-12-16T07:30:00.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.blob_compression_type: NoCompression 2023-12-16T07:30:00.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-12-16T07:30:00.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-12-16T07:30:00.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-16T07:30:00.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2023-12-16T07:30:00.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: [db/version_set.cc:4083] Creating manifest 7 2023-12-16T07:30:00.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702711800670416, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-12-16T07:30:00.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 4 rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2023-12-16T07:30:00.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.664+0000 7fe9e53df880 3 rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-16T07:30:00.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.668+0000 7fe9e53df880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702711800671107, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1702711800, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "ae593167-e7ee-44bc-a4a6-2e1aa089ddc6", "db_session_id": "XROYWWR2CCF9UGV729AJ"}} 2023-12-16T07:30:00.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.668+0000 7fe9e53df880 4 rocksdb: [db/version_set.cc:4083] Creating manifest 9 2023-12-16T07:30:00.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.668+0000 7fe9e53df880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702711800672015, "job": 1, "event": "recovery_finished"} 2023-12-16T07:30:00.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.668+0000 7fe9e53df880 4 rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-12-16T07:30:00.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.668+0000 7fe9e53df880 4 rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x555a26d76a80 2023-12-16T07:30:00.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.668+0000 7fe9e53df880 4 rocksdb: DB pointer 0x555a26d86000 2023-12-16T07:30:00.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.668+0000 7fe9d5167700 4 rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2023-12-16T07:30:00.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.668+0000 7fe9d5167700 4 rocksdb: [db/db_impl/db_impl.cc:903] 2023-12-16T07:30:00.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: ** DB Stats ** 2023-12-16T07:30:00.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:30:00.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-16T07:30:00.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-16T07:30:00.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T07:30:00.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-16T07:30:00.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-16T07:30:00.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T07:30:00.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: ** Compaction Stats [default] ** 2023-12-16T07:30:00.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:30:00.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:30:00.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.2 0.00 0.00 1 0.001 0 0 2023-12-16T07:30:00.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.2 0.00 0.00 1 0.001 0 0 2023-12-16T07:30:00.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.2 0.00 0.00 1 0.001 0 0 2023-12-16T07:30:00.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: ** Compaction Stats [default] ** 2023-12-16T07:30:00.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:30:00.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:30:00.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.2 0.00 0.00 1 0.001 0 0 2023-12-16T07:30:00.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:30:00.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T07:30:00.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T07:30:00.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T07:30:00.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T07:30:00.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: AddFile(Keys): cumulative 0, interval 0 2023-12-16T07:30:00.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Cumulative compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:30:00.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Interval compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:30:00.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T07:30:00.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: ** File Read Latency Histogram By Level [default] ** 2023-12-16T07:30:00.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: ** Compaction Stats [default] ** 2023-12-16T07:30:00.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:30:00.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:30:00.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.2 0.00 0.00 1 0.001 0 0 2023-12-16T07:30:00.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.2 0.00 0.00 1 0.001 0 0 2023-12-16T07:30:00.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-16T07:30:00.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: ** Compaction Stats [default] ** 2023-12-16T07:30:00.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:30:00.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:30:00.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.2 0.00 0.00 1 0.001 0 0 2023-12-16T07:30:00.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:30:00.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T07:30:00.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T07:30:00.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T07:30:00.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T07:30:00.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: AddFile(Keys): cumulative 0, interval 0 2023-12-16T07:30:00.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Cumulative compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:30:00.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:30:00.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T07:30:00.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: ** File Read Latency Histogram By Level [default] ** 2023-12-16T07:30:00.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.668+0000 7fe9e53df880 0 mon.c does not exist in monmap, will attempt to join an existing cluster 2023-12-16T07:30:00.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.668+0000 7fe9e53df880 0 using public_addrv [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-12-16T07:30:00.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.672+0000 7fe9e53df880 0 starting mon.c rank -1 at public addrs [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] at bind addrs [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:30:00.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.672+0000 7fe9e53df880 1 mon.c@-1(???) e0 preinit fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:30:00.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.692+0000 7fe9d816d700 0 mon.c@-1(synchronizing).mds e1 new map 2023-12-16T07:30:00.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.692+0000 7fe9d816d700 0 mon.c@-1(synchronizing).mds e1 print_map 2023-12-16T07:30:00.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: e1 2023-12-16T07:30:00.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: enable_multiple, ever_enabled_multiple: 1,1 2023-12-16T07:30:00.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T07:30:00.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: legacy client fscid: -1 2023-12-16T07:30:00.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: 2023-12-16T07:30:00.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: No filesystems configured 2023-12-16T07:30:00.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.692+0000 7fe9d816d700 1 mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-12-16T07:30:00.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.692+0000 7fe9d816d700 1 mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-12-16T07:30:00.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.692+0000 7fe9d816d700 1 mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-12-16T07:30:00.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.692+0000 7fe9d816d700 1 mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-12-16T07:30:00.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.692+0000 7fe9d816d700 1 mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-12-16T07:30:00.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.692+0000 7fe9d816d700 1 mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-12-16T07:30:00.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.692+0000 7fe9d816d700 0 mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-16T07:30:00.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.692+0000 7fe9d816d700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T07:30:00.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.692+0000 7fe9d816d700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T07:30:00.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.692+0000 7fe9d816d700 0 mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T07:30:00.840 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cluster 2023-12-16T07:29:45.319642+0000 mgr.y (mgr.14152) 19 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:00.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:46.503886+0000 mgr.y (mgr.14152) 20 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T07:30:00.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cluster 2023-12-16T07:29:47.320079+0000 mgr.y (mgr.14152) 21 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:00.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:49.172682+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:00.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:49.179126+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:00.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:49.180630+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:30:00.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:49.186529+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:00.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cluster 2023-12-16T07:29:49.320512+0000 mgr.y (mgr.14152) 22 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:00.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:50.369279+0000 mon.a (mon.0) 119 : audit [INF] from='client.? 172.21.15.7:0/2702297157' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-16T07:30:00.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:51.194448+0000 mon.a (mon.0) 120 : audit [INF] from='client.? 172.21.15.7:0/2702297157' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-16T07:30:00.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cluster 2023-12-16T07:29:51.194584+0000 mon.a (mon.0) 121 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-16T07:30:00.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cluster 2023-12-16T07:29:51.320931+0000 mgr.y (mgr.14152) 23 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:00.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cluster 2023-12-16T07:29:53.321430+0000 mgr.y (mgr.14152) 24 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:00.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:54.767707+0000 mgr.y (mgr.14152) 25 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi169:172.21.15.169=b", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:30:00.841 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cephadm 2023-12-16T07:29:54.772975+0000 mgr.y (mgr.14152) 26 : cephadm [INF] Saving service mon spec with placement smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi169:172.21.15.169=b;count:3 2023-12-16T07:30:00.842 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:54.778493+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:00.842 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:54.817011+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:00.842 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:54.821740+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:00.842 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:54.823680+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:00.842 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cephadm 2023-12-16T07:29:54.826380+0000 mgr.y (mgr.14152) 27 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:30:00.842 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cephadm 2023-12-16T07:29:54.940106+0000 mgr.y (mgr.14152) 28 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:30:00.842 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:55.164388+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:00.842 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:55.170721+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:00.842 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:55.174360+0000 mon.a (mon.0) 128 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:30:00.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:55.176049+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:00.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cephadm 2023-12-16T07:29:55.177393+0000 mgr.y (mgr.14152) 29 : cephadm [INF] Deploying daemon mon.c on smithi007 2023-12-16T07:30:00.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cluster 2023-12-16T07:29:55.322003+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:00.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cluster 2023-12-16T07:29:57.322491+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:00.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:58.328677+0000 mon.a (mon.0) 130 : audit [DBG] from='client.? 172.21.15.169:0/2811537117' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T07:30:00.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cluster 2023-12-16T07:29:59.322809+0000 mgr.y (mgr.14152) 32 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:00.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:59.632463+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:00.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:59.633755+0000 mon.a (mon.0) 132 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:30:00.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: audit 2023-12-16T07:29:59.634670+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:00.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: cephadm 2023-12-16T07:29:59.635673+0000 mgr.y (mgr.14152) 33 : cephadm [INF] Deploying daemon mon.b on smithi169 2023-12-16T07:30:00.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.696+0000 7fe9d816d700 1 mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-12-16T07:30:00.843 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:00 smithi007 bash[30436]: debug 2023-12-16T07:30:00.696+0000 7fe9d816d700 10 mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-12-16T07:30:05.411 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:05 smithi169 systemd[1]: Started Ceph mon.b for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:30:06.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: audit 2023-12-16T07:30:00.709337+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:30:06.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: cluster 2023-12-16T07:30:00.709641+0000 mon.a (mon.0) 136 : cluster [INF] mon.a calling monitor election 2023-12-16T07:30:06.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: audit 2023-12-16T07:30:00.710938+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: cluster 2023-12-16T07:30:01.323196+0000 mgr.y (mgr.14152) 34 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:06.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: audit 2023-12-16T07:30:01.704946+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: audit 2023-12-16T07:30:02.704915+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: cluster 2023-12-16T07:30:02.707639+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2023-12-16T07:30:06.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: cluster 2023-12-16T07:30:03.323592+0000 mgr.y (mgr.14152) 35 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:06.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: audit 2023-12-16T07:30:03.705161+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: audit 2023-12-16T07:30:04.705568+0000 mon.a (mon.0) 141 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: audit 2023-12-16T07:30:05.705422+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: cluster 2023-12-16T07:30:05.714169+0000 mon.a (mon.0) 143 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-12-16T07:30:06.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: cluster 2023-12-16T07:30:05.720437+0000 mon.a (mon.0) 144 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2023-12-16T07:30:06.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: cluster 2023-12-16T07:30:05.720618+0000 mon.a (mon.0) 145 : cluster [DBG] fsmap 2023-12-16T07:30:06.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: cluster 2023-12-16T07:30:05.720715+0000 mon.a (mon.0) 146 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-16T07:30:06.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: cluster 2023-12-16T07:30:05.721042+0000 mon.a (mon.0) 147 : cluster [DBG] mgrmap e13: y(active, since 50s) 2023-12-16T07:30:06.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: cluster 2023-12-16T07:30:05.728439+0000 mon.a (mon.0) 148 : cluster [INF] overall HEALTH_OK 2023-12-16T07:30:06.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: audit 2023-12-16T07:30:05.733242+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:06.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: audit 2023-12-16T07:30:05.735851+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:06.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: audit 2023-12-16T07:30:05.738308+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:06.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[25009]: audit 2023-12-16T07:30:05.739944+0000 mon.a (mon.0) 152 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:06.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: audit 2023-12-16T07:30:00.709337+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:30:06.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: cluster 2023-12-16T07:30:00.709641+0000 mon.a (mon.0) 136 : cluster [INF] mon.a calling monitor election 2023-12-16T07:30:06.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: audit 2023-12-16T07:30:00.710938+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: cluster 2023-12-16T07:30:01.323196+0000 mgr.y (mgr.14152) 34 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:06.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: audit 2023-12-16T07:30:01.704946+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: audit 2023-12-16T07:30:02.704915+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: cluster 2023-12-16T07:30:02.707639+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2023-12-16T07:30:06.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: cluster 2023-12-16T07:30:03.323592+0000 mgr.y (mgr.14152) 35 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:06.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: audit 2023-12-16T07:30:03.705161+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: audit 2023-12-16T07:30:04.705568+0000 mon.a (mon.0) 141 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: audit 2023-12-16T07:30:05.705422+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: cluster 2023-12-16T07:30:05.714169+0000 mon.a (mon.0) 143 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-12-16T07:30:06.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: cluster 2023-12-16T07:30:05.720437+0000 mon.a (mon.0) 144 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2023-12-16T07:30:06.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: cluster 2023-12-16T07:30:05.720618+0000 mon.a (mon.0) 145 : cluster [DBG] fsmap 2023-12-16T07:30:06.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: cluster 2023-12-16T07:30:05.720715+0000 mon.a (mon.0) 146 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-16T07:30:06.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: cluster 2023-12-16T07:30:05.721042+0000 mon.a (mon.0) 147 : cluster [DBG] mgrmap e13: y(active, since 50s) 2023-12-16T07:30:06.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: cluster 2023-12-16T07:30:05.728439+0000 mon.a (mon.0) 148 : cluster [INF] overall HEALTH_OK 2023-12-16T07:30:06.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: audit 2023-12-16T07:30:05.733242+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:06.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: audit 2023-12-16T07:30:05.735851+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:06.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: audit 2023-12-16T07:30:05.738308+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:06.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:05 smithi007 bash[30436]: audit 2023-12-16T07:30:05.739944+0000 mon.a (mon.0) 152 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.378+0000 7f4b49c57880 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-16T07:30:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.378+0000 7f4b49c57880 0 ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 7 2023-12-16T07:30:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.378+0000 7f4b49c57880 0 pidfile_write: ignore empty --pid-file 2023-12-16T07:30:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.382+0000 7f4b49c57880 0 load: jerasure load: lrc 2023-12-16T07:30:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: RocksDB version: 6.15.5 2023-12-16T07:30:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-16T07:30:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Compile date Apr 18 2022 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: DB SUMMARY 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: DB Session ID: IF3B6POQHVKBOV6OT24G 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: CURRENT file: CURRENT 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: IDENTITY file: IDENTITY 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.error_if_exists: 0 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.create_if_missing: 0 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.paranoid_checks: 1 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.env: 0x55f14bfcd860 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.fs: Posix File System 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.info_log: 0x55f14e7a9e00 2023-12-16T07:30:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-16T07:30:06.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.statistics: (nil) 2023-12-16T07:30:06.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.use_fsync: 0 2023-12-16T07:30:06.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_log_file_size: 0 2023-12-16T07:30:06.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-16T07:30:06.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-16T07:30:06.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-16T07:30:06.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-16T07:30:06.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.allow_fallocate: 1 2023-12-16T07:30:06.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-16T07:30:06.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-16T07:30:06.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.use_direct_reads: 0 2023-12-16T07:30:06.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-16T07:30:06.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.create_missing_column_families: 0 2023-12-16T07:30:06.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.db_log_dir: 2023-12-16T07:30:06.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2023-12-16T07:30:06.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-16T07:30:06.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-16T07:30:06.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-16T07:30:06.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-16T07:30:06.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-16T07:30:06.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-16T07:30:06.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.advise_random_on_open: 1 2023-12-16T07:30:06.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-16T07:30:06.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.write_buffer_manager: 0x55f14e89a2d0 2023-12-16T07:30:06.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-16T07:30:06.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-16T07:30:06.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-16T07:30:06.687 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-16T07:30:06.687 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.rate_limiter: (nil) 2023-12-16T07:30:06.687 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-16T07:30:06.687 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-16T07:30:06.687 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-16T07:30:06.688 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-16T07:30:06.688 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.unordered_write: 0 2023-12-16T07:30:06.688 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-16T07:30:06.688 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-16T07:30:06.689 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-16T07:30:06.689 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-16T07:30:06.689 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.row_cache: None 2023-12-16T07:30:06.689 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.wal_filter: None 2023-12-16T07:30:06.689 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-16T07:30:06.689 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-16T07:30:06.689 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.preserve_deletes: 0 2023-12-16T07:30:06.689 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.two_write_queues: 0 2023-12-16T07:30:06.689 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.manual_wal_flush: 0 2023-12-16T07:30:06.689 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.atomic_flush: 0 2023-12-16T07:30:06.690 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-16T07:30:06.690 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-16T07:30:06.690 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-16T07:30:06.690 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.log_readahead_size: 0 2023-12-16T07:30:06.690 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-12-16T07:30:06.690 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.best_efforts_recovery: 0 2023-12-16T07:30:06.690 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-12-16T07:30:06.690 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-12-16T07:30:06.690 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.allow_data_in_errors: 0 2023-12-16T07:30:06.690 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.db_host_id: __hostname__ 2023-12-16T07:30:06.690 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_background_jobs: 2 2023-12-16T07:30:06.690 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_background_compactions: -1 2023-12-16T07:30:06.690 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_subcompactions: 1 2023-12-16T07:30:06.691 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-16T07:30:06.691 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-16T07:30:06.691 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-16T07:30:06.691 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_total_wal_size: 0 2023-12-16T07:30:06.691 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-16T07:30:06.691 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-16T07:30:06.691 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-16T07:30:06.691 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-16T07:30:06.691 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_open_files: -1 2023-12-16T07:30:06.691 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.bytes_per_sync: 0 2023-12-16T07:30:06.691 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-16T07:30:06.691 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-16T07:30:06.691 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-16T07:30:06.692 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_background_flushes: -1 2023-12-16T07:30:06.692 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Compression algorithms supported: 2023-12-16T07:30:06.692 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-16T07:30:06.692 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: kZSTD supported: 0 2023-12-16T07:30:06.692 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: kXpressCompression supported: 0 2023-12-16T07:30:06.692 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-16T07:30:06.692 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: kLZ4Compression supported: 1 2023-12-16T07:30:06.692 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: kBZip2Compression supported: 0 2023-12-16T07:30:06.692 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: kZlibCompression supported: 1 2023-12-16T07:30:06.692 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: kSnappyCompression supported: 1 2023-12-16T07:30:06.692 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-16T07:30:06.693 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2023-12-16T07:30:06.693 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2023-12-16T07:30:06.693 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-16T07:30:06.693 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.merge_operator: 2023-12-16T07:30:06.693 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compaction_filter: None 2023-12-16T07:30:06.693 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compaction_filter_factory: None 2023-12-16T07:30:06.693 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.sst_partitioner_factory: None 2023-12-16T07:30:06.693 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-16T07:30:06.693 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-16T07:30:06.693 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f14e777d00) 2023-12-16T07:30:06.693 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cache_index_and_filter_blocks: 1 2023-12-16T07:30:06.693 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-16T07:30:06.694 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-16T07:30:06.694 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: pin_top_level_index_and_filter: 1 2023-12-16T07:30:06.694 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: index_type: 0 2023-12-16T07:30:06.694 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: data_block_index_type: 0 2023-12-16T07:30:06.694 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: index_shortening: 1 2023-12-16T07:30:06.694 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: data_block_hash_table_util_ratio: 0.750000 2023-12-16T07:30:06.694 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: hash_index_allow_collision: 1 2023-12-16T07:30:06.694 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: checksum: 1 2023-12-16T07:30:06.694 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: no_block_cache: 0 2023-12-16T07:30:06.694 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: block_cache: 0x55f14e7de170 2023-12-16T07:30:06.694 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: block_cache_name: BinnedLRUCache 2023-12-16T07:30:06.694 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: block_cache_options: 2023-12-16T07:30:06.695 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: capacity : 536870912 2023-12-16T07:30:06.695 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: num_shard_bits : 4 2023-12-16T07:30:06.695 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: strict_capacity_limit : 0 2023-12-16T07:30:06.695 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: high_pri_pool_ratio: 0.000 2023-12-16T07:30:06.695 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: block_cache_compressed: (nil) 2023-12-16T07:30:06.695 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: persistent_cache: (nil) 2023-12-16T07:30:06.695 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: block_size: 4096 2023-12-16T07:30:06.695 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: block_size_deviation: 10 2023-12-16T07:30:06.695 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: block_restart_interval: 16 2023-12-16T07:30:06.695 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: index_block_restart_interval: 1 2023-12-16T07:30:06.695 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: metadata_block_size: 4096 2023-12-16T07:30:06.696 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: partition_filters: 0 2023-12-16T07:30:06.696 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: use_delta_encoding: 1 2023-12-16T07:30:06.696 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-16T07:30:06.696 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: whole_key_filtering: 1 2023-12-16T07:30:06.696 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: verify_compression: 0 2023-12-16T07:30:06.696 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: read_amp_bytes_per_bit: 0 2023-12-16T07:30:06.696 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: format_version: 4 2023-12-16T07:30:06.696 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: enable_index_compression: 1 2023-12-16T07:30:06.696 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: block_align: 0 2023-12-16T07:30:06.696 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-16T07:30:06.696 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-16T07:30:06.697 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compression: NoCompression 2023-12-16T07:30:06.697 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-16T07:30:06.697 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-16T07:30:06.697 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-16T07:30:06.697 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.num_levels: 7 2023-12-16T07:30:06.697 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-16T07:30:06.697 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-16T07:30:06.697 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-16T07:30:06.697 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-16T07:30:06.697 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-16T07:30:06.697 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-16T07:30:06.698 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-16T07:30:06.698 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-16T07:30:06.698 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-12-16T07:30:06.698 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-16T07:30:06.698 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-16T07:30:06.698 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compression_opts.level: 32767 2023-12-16T07:30:06.698 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-16T07:30:06.698 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-16T07:30:06.698 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-16T07:30:06.698 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-12-16T07:30:06.698 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compression_opts.enabled: false 2023-12-16T07:30:06.698 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-16T07:30:06.699 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-16T07:30:06.699 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-16T07:30:06.699 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-16T07:30:06.699 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-16T07:30:06.699 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-16T07:30:06.699 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-16T07:30:06.699 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-16T07:30:06.699 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-16T07:30:06.699 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-16T07:30:06.699 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-16T07:30:06.700 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-16T07:30:06.700 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-16T07:30:06.700 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-16T07:30:06.700 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-16T07:30:06.700 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-16T07:30:06.700 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-16T07:30:06.700 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.arena_block_size: 4194304 2023-12-16T07:30:06.700 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-16T07:30:06.700 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-16T07:30:06.700 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-16T07:30:06.701 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-16T07:30:06.701 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-16T07:30:06.701 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-16T07:30:06.701 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-16T07:30:06.701 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-16T07:30:06.701 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-16T07:30:06.701 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-16T07:30:06.701 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-16T07:30:06.701 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-16T07:30:06.701 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-16T07:30:06.702 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-16T07:30:06.702 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.table_properties_collectors: 2023-12-16T07:30:06.702 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.inplace_update_support: 0 2023-12-16T07:30:06.702 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-16T07:30:06.702 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-16T07:30:06.702 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-16T07:30:06.702 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-16T07:30:06.702 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.bloom_locality: 0 2023-12-16T07:30:06.702 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.max_successive_merges: 0 2023-12-16T07:30:06.702 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-16T07:30:06.703 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-16T07:30:06.703 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.force_consistency_checks: 1 2023-12-16T07:30:06.703 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-16T07:30:06.703 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.ttl: 2592000 2023-12-16T07:30:06.703 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-16T07:30:06.703 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.enable_blob_files: false 2023-12-16T07:30:06.703 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.min_blob_size: 0 2023-12-16T07:30:06.703 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.blob_file_size: 268435456 2023-12-16T07:30:06.703 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.blob_compression_type: NoCompression 2023-12-16T07:30:06.703 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-12-16T07:30:06.703 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-12-16T07:30:06.703 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-16T07:30:06.703 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2023-12-16T07:30:06.703 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: [db/version_set.cc:4083] Creating manifest 7 2023-12-16T07:30:06.704 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702711806392689, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-12-16T07:30:06.704 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2023-12-16T07:30:06.704 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 3 rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-16T07:30:06.704 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702711806393107, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1702711806, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "b127e6f1-4aeb-4f74-a2b3-9a9e50d388de", "db_session_id": "IF3B6POQHVKBOV6OT24G"}} 2023-12-16T07:30:06.704 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: [db/version_set.cc:4083] Creating manifest 9 2023-12-16T07:30:06.704 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.386+0000 7f4b49c57880 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702711806393883, "job": 1, "event": "recovery_finished"} 2023-12-16T07:30:06.704 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.390+0000 7f4b49c57880 4 rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-12-16T07:30:06.704 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.390+0000 7f4b49c57880 4 rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55f14e7c4a80 2023-12-16T07:30:06.704 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.390+0000 7f4b49c57880 4 rocksdb: DB pointer 0x55f14e7d4000 2023-12-16T07:30:06.704 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.390+0000 7f4b399df700 4 rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2023-12-16T07:30:06.705 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.390+0000 7f4b399df700 4 rocksdb: [db/db_impl/db_impl.cc:903] 2023-12-16T07:30:06.705 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: ** DB Stats ** 2023-12-16T07:30:06.705 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:30:06.705 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-16T07:30:06.705 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-16T07:30:06.705 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T07:30:06.705 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-16T07:30:06.705 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-16T07:30:06.705 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T07:30:06.705 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: ** Compaction Stats [default] ** 2023-12-16T07:30:06.705 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:30:06.705 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:30:06.705 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 2023-12-16T07:30:06.706 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 2023-12-16T07:30:06.706 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 2023-12-16T07:30:06.706 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: ** Compaction Stats [default] ** 2023-12-16T07:30:06.706 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:30:06.706 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:30:06.706 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 2023-12-16T07:30:06.706 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:30:06.706 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T07:30:06.706 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T07:30:06.706 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T07:30:06.706 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T07:30:06.706 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: AddFile(Keys): cumulative 0, interval 0 2023-12-16T07:30:06.706 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Cumulative compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:30:06.707 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Interval compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:30:06.707 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T07:30:06.707 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: ** File Read Latency Histogram By Level [default] ** 2023-12-16T07:30:06.707 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: ** Compaction Stats [default] ** 2023-12-16T07:30:06.707 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:30:06.707 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:30:06.707 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 2023-12-16T07:30:06.707 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 2023-12-16T07:30:06.707 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-16T07:30:06.707 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: ** Compaction Stats [default] ** 2023-12-16T07:30:06.707 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-16T07:30:06.707 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:30:06.707 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 2023-12-16T07:30:06.708 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:30:06.708 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T07:30:06.708 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T07:30:06.708 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T07:30:06.708 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T07:30:06.708 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: AddFile(Keys): cumulative 0, interval 0 2023-12-16T07:30:06.708 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Cumulative compaction: 0.00 GB write, 0.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:30:06.708 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:30:06.708 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T07:30:06.708 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: ** File Read Latency Histogram By Level [default] ** 2023-12-16T07:30:06.708 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.390+0000 7f4b49c57880 0 mon.b does not exist in monmap, will attempt to join an existing cluster 2023-12-16T07:30:06.708 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.390+0000 7f4b49c57880 0 using public_addr v2:172.21.15.169:0/0 -> [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] 2023-12-16T07:30:06.708 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.394+0000 7f4b49c57880 0 starting mon.b rank -1 at public addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] at bind addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:30:06.708 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.394+0000 7f4b49c57880 1 mon.b@-1(???) e0 preinit fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:30:06.709 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.418+0000 7f4b3c9e5700 0 mon.b@-1(synchronizing).mds e1 new map 2023-12-16T07:30:06.709 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.418+0000 7f4b3c9e5700 0 mon.b@-1(synchronizing).mds e1 print_map 2023-12-16T07:30:06.709 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: e1 2023-12-16T07:30:06.709 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: enable_multiple, ever_enabled_multiple: 1,1 2023-12-16T07:30:06.709 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T07:30:06.709 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: legacy client fscid: -1 2023-12-16T07:30:06.709 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: 2023-12-16T07:30:06.709 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: No filesystems configured 2023-12-16T07:30:06.709 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.418+0000 7f4b3c9e5700 1 mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-12-16T07:30:06.709 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.418+0000 7f4b3c9e5700 1 mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-12-16T07:30:06.709 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.418+0000 7f4b3c9e5700 1 mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-12-16T07:30:06.709 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.418+0000 7f4b3c9e5700 1 mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-12-16T07:30:06.709 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.418+0000 7f4b3c9e5700 1 mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-12-16T07:30:06.709 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.418+0000 7f4b3c9e5700 1 mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.418+0000 7f4b3c9e5700 0 mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.418+0000 7f4b3c9e5700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.418+0000 7f4b3c9e5700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.418+0000 7f4b3c9e5700 0 mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:29:45.319642+0000 mgr.y (mgr.14152) 19 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:46.503886+0000 mgr.y (mgr.14152) 20 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:29:47.320079+0000 mgr.y (mgr.14152) 21 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:49.172682+0000 mon.a (mon.0) 115 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:49.179126+0000 mon.a (mon.0) 116 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:49.180630+0000 mon.a (mon.0) 117 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi169", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:49.186529+0000 mon.a (mon.0) 118 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:29:49.320512+0000 mgr.y (mgr.14152) 22 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:50.369279+0000 mon.a (mon.0) 119 : audit [INF] from='client.? 172.21.15.7:0/2702297157' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:51.194448+0000 mon.a (mon.0) 120 : audit [INF] from='client.? 172.21.15.7:0/2702297157' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-16T07:30:06.710 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:29:51.194584+0000 mon.a (mon.0) 121 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-16T07:30:06.711 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:29:51.320931+0000 mgr.y (mgr.14152) 23 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:06.711 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:29:53.321430+0000 mgr.y (mgr.14152) 24 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:06.711 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:54.767707+0000 mgr.y (mgr.14152) 25 : audit [DBG] from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi169:172.21.15.169=b", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:30:06.711 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cephadm 2023-12-16T07:29:54.772975+0000 mgr.y (mgr.14152) 26 : cephadm [INF] Saving service mon spec with placement smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi169:172.21.15.169=b;count:3 2023-12-16T07:30:06.711 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:54.778493+0000 mon.a (mon.0) 122 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:06.711 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:54.817011+0000 mon.a (mon.0) 123 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:06.711 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:54.821740+0000 mon.a (mon.0) 124 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:06.711 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:54.823680+0000 mon.a (mon.0) 125 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:06.711 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cephadm 2023-12-16T07:29:54.826380+0000 mgr.y (mgr.14152) 27 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:30:06.712 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cephadm 2023-12-16T07:29:54.940106+0000 mgr.y (mgr.14152) 28 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:30:06.712 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:55.164388+0000 mon.a (mon.0) 126 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:06.712 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:55.170721+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:06.712 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:55.174360+0000 mon.a (mon.0) 128 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:30:06.712 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:55.176049+0000 mon.a (mon.0) 129 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:06.712 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cephadm 2023-12-16T07:29:55.177393+0000 mgr.y (mgr.14152) 29 : cephadm [INF] Deploying daemon mon.c on smithi007 2023-12-16T07:30:06.712 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:29:55.322003+0000 mgr.y (mgr.14152) 30 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:06.712 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:29:57.322491+0000 mgr.y (mgr.14152) 31 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:06.712 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:58.328677+0000 mon.a (mon.0) 130 : audit [DBG] from='client.? 172.21.15.169:0/2811537117' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T07:30:06.712 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:29:59.322809+0000 mgr.y (mgr.14152) 32 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:06.712 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:59.632463+0000 mon.a (mon.0) 131 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:06.712 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:59.633755+0000 mon.a (mon.0) 132 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:30:06.712 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:29:59.634670+0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:06.712 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cephadm 2023-12-16T07:29:59.635673+0000 mgr.y (mgr.14152) 33 : cephadm [INF] Deploying daemon mon.b on smithi169 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:30:00.709337+0000 mon.a (mon.0) 135 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:30:00.709641+0000 mon.a (mon.0) 136 : cluster [INF] mon.a calling monitor election 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:30:00.710938+0000 mon.a (mon.0) 137 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:30:01.323196+0000 mgr.y (mgr.14152) 34 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:30:01.704946+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:30:02.704915+0000 mon.a (mon.0) 139 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:30:02.707639+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:30:03.323592+0000 mgr.y (mgr.14152) 35 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:30:03.705161+0000 mon.a (mon.0) 140 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:30:04.705568+0000 mon.a (mon.0) 141 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:30:05.705422+0000 mon.a (mon.0) 142 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:30:05.714169+0000 mon.a (mon.0) 143 : cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:30:05.720437+0000 mon.a (mon.0) 144 : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:30:05.720618+0000 mon.a (mon.0) 145 : cluster [DBG] fsmap 2023-12-16T07:30:06.713 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:30:05.720715+0000 mon.a (mon.0) 146 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-16T07:30:06.714 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:30:05.721042+0000 mon.a (mon.0) 147 : cluster [DBG] mgrmap e13: y(active, since 50s) 2023-12-16T07:30:06.714 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: cluster 2023-12-16T07:30:05.728439+0000 mon.a (mon.0) 148 : cluster [INF] overall HEALTH_OK 2023-12-16T07:30:06.714 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:30:05.733242+0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:06.714 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:30:05.735851+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:06.714 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:30:05.738308+0000 mon.a (mon.0) 151 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:06.714 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: audit 2023-12-16T07:30:05.739944+0000 mon.a (mon.0) 152 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:06.714 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.422+0000 7f4b3c9e5700 1 mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-12-16T07:30:06.714 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:06 smithi169 bash[25083]: debug 2023-12-16T07:30:06.426+0000 7f4b3c9e5700 10 mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-12-16T07:30:11.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: audit 2023-12-16T07:30:06.440323+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:30:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: cluster 2023-12-16T07:30:06.440659+0000 mon.a (mon.0) 155 : cluster [INF] mon.a calling monitor election 2023-12-16T07:30:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: cluster 2023-12-16T07:30:06.441647+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2023-12-16T07:30:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: audit 2023-12-16T07:30:06.442452+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: audit 2023-12-16T07:30:06.442738+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: cluster 2023-12-16T07:30:07.324516+0000 mgr.y (mgr.14152) 37 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: audit 2023-12-16T07:30:07.433312+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: audit 2023-12-16T07:30:08.433478+0000 mon.a (mon.0) 159 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: cluster 2023-12-16T07:30:08.438072+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2023-12-16T07:30:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: cluster 2023-12-16T07:30:09.325002+0000 mgr.y (mgr.14152) 38 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:11.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: audit 2023-12-16T07:30:09.433576+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: audit 2023-12-16T07:30:10.433764+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: audit 2023-12-16T07:30:11.433707+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: cluster 2023-12-16T07:30:11.462725+0000 mon.a (mon.0) 163 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-12-16T07:30:11.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: cluster 2023-12-16T07:30:11.469599+0000 mon.a (mon.0) 164 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2023-12-16T07:30:11.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: cluster 2023-12-16T07:30:11.469715+0000 mon.a (mon.0) 165 : cluster [DBG] fsmap 2023-12-16T07:30:11.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: cluster 2023-12-16T07:30:11.469772+0000 mon.a (mon.0) 166 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-16T07:30:11.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: cluster 2023-12-16T07:30:11.470100+0000 mon.a (mon.0) 167 : cluster [DBG] mgrmap e13: y(active, since 56s) 2023-12-16T07:30:11.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: cluster 2023-12-16T07:30:11.477495+0000 mon.a (mon.0) 168 : cluster [INF] overall HEALTH_OK 2023-12-16T07:30:11.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: audit 2023-12-16T07:30:11.482385+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:11.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: audit 2023-12-16T07:30:11.486718+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:11.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[25009]: audit 2023-12-16T07:30:11.491485+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:11.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: audit 2023-12-16T07:30:06.440323+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:30:11.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: cluster 2023-12-16T07:30:06.440659+0000 mon.a (mon.0) 155 : cluster [INF] mon.a calling monitor election 2023-12-16T07:30:11.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: cluster 2023-12-16T07:30:06.441647+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2023-12-16T07:30:11.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: audit 2023-12-16T07:30:06.442452+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: audit 2023-12-16T07:30:06.442738+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:11.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: cluster 2023-12-16T07:30:07.324516+0000 mgr.y (mgr.14152) 37 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:11.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: audit 2023-12-16T07:30:07.433312+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: audit 2023-12-16T07:30:08.433478+0000 mon.a (mon.0) 159 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: cluster 2023-12-16T07:30:08.438072+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2023-12-16T07:30:11.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: cluster 2023-12-16T07:30:09.325002+0000 mgr.y (mgr.14152) 38 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:11.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: audit 2023-12-16T07:30:09.433576+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: audit 2023-12-16T07:30:10.433764+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: audit 2023-12-16T07:30:11.433707+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: cluster 2023-12-16T07:30:11.462725+0000 mon.a (mon.0) 163 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-12-16T07:30:11.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: cluster 2023-12-16T07:30:11.469599+0000 mon.a (mon.0) 164 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2023-12-16T07:30:11.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: cluster 2023-12-16T07:30:11.469715+0000 mon.a (mon.0) 165 : cluster [DBG] fsmap 2023-12-16T07:30:11.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: cluster 2023-12-16T07:30:11.469772+0000 mon.a (mon.0) 166 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-16T07:30:11.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: cluster 2023-12-16T07:30:11.470100+0000 mon.a (mon.0) 167 : cluster [DBG] mgrmap e13: y(active, since 56s) 2023-12-16T07:30:11.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: cluster 2023-12-16T07:30:11.477495+0000 mon.a (mon.0) 168 : cluster [INF] overall HEALTH_OK 2023-12-16T07:30:11.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: audit 2023-12-16T07:30:11.482385+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:11.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: audit 2023-12-16T07:30:11.486718+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:11.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:11 smithi007 bash[30436]: audit 2023-12-16T07:30:11.491485+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:11.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: audit 2023-12-16T07:30:06.440323+0000 mon.a (mon.0) 154 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:30:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: cluster 2023-12-16T07:30:06.440659+0000 mon.a (mon.0) 155 : cluster [INF] mon.a calling monitor election 2023-12-16T07:30:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: cluster 2023-12-16T07:30:06.441647+0000 mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2023-12-16T07:30:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: audit 2023-12-16T07:30:06.442452+0000 mon.a (mon.0) 156 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: audit 2023-12-16T07:30:06.442738+0000 mon.a (mon.0) 157 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:30:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: cluster 2023-12-16T07:30:07.324516+0000 mgr.y (mgr.14152) 37 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: audit 2023-12-16T07:30:07.433312+0000 mon.a (mon.0) 158 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: audit 2023-12-16T07:30:08.433478+0000 mon.a (mon.0) 159 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: cluster 2023-12-16T07:30:08.438072+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2023-12-16T07:30:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: cluster 2023-12-16T07:30:09.325002+0000 mgr.y (mgr.14152) 38 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: audit 2023-12-16T07:30:09.433576+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: audit 2023-12-16T07:30:10.433764+0000 mon.a (mon.0) 161 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: audit 2023-12-16T07:30:11.433707+0000 mon.a (mon.0) 162 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: cluster 2023-12-16T07:30:11.462725+0000 mon.a (mon.0) 163 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-12-16T07:30:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: cluster 2023-12-16T07:30:11.469599+0000 mon.a (mon.0) 164 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2023-12-16T07:30:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: cluster 2023-12-16T07:30:11.469715+0000 mon.a (mon.0) 165 : cluster [DBG] fsmap 2023-12-16T07:30:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: cluster 2023-12-16T07:30:11.469772+0000 mon.a (mon.0) 166 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-16T07:30:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: cluster 2023-12-16T07:30:11.470100+0000 mon.a (mon.0) 167 : cluster [DBG] mgrmap e13: y(active, since 56s) 2023-12-16T07:30:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: cluster 2023-12-16T07:30:11.477495+0000 mon.a (mon.0) 168 : cluster [INF] overall HEALTH_OK 2023-12-16T07:30:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: audit 2023-12-16T07:30:11.482385+0000 mon.a (mon.0) 169 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: audit 2023-12-16T07:30:11.486718+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:11 smithi169 bash[25083]: audit 2023-12-16T07:30:11.491485+0000 mon.a (mon.0) 171 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:12.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[25009]: cluster 2023-12-16T07:30:11.325439+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:12.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[25009]: cephadm 2023-12-16T07:30:11.483525+0000 mgr.y (mgr.14152) 40 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:30:12.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[25009]: cephadm 2023-12-16T07:30:11.487775+0000 mgr.y (mgr.14152) 41 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:30:12.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[25009]: audit 2023-12-16T07:30:11.625332+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:12.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[25009]: audit 2023-12-16T07:30:11.633956+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:12.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[25009]: audit 2023-12-16T07:30:11.639046+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:12.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[25009]: cephadm 2023-12-16T07:30:11.640804+0000 mgr.y (mgr.14152) 42 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2023-12-16T07:30:12.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[25009]: audit 2023-12-16T07:30:11.641211+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:30:12.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[25009]: audit 2023-12-16T07:30:11.642402+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:30:12.932 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[25009]: audit 2023-12-16T07:30:11.643563+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:12.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[25009]: cephadm 2023-12-16T07:30:11.644927+0000 mgr.y (mgr.14152) 43 : cephadm [INF] Reconfiguring daemon mon.a on smithi007 2023-12-16T07:30:12.933 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[25009]: audit 2023-12-16T07:30:12.434328+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:12.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[30436]: cluster 2023-12-16T07:30:11.325439+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:12.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[30436]: cephadm 2023-12-16T07:30:11.483525+0000 mgr.y (mgr.14152) 40 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:30:12.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[30436]: cephadm 2023-12-16T07:30:11.487775+0000 mgr.y (mgr.14152) 41 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:30:12.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[30436]: audit 2023-12-16T07:30:11.625332+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:12.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[30436]: audit 2023-12-16T07:30:11.633956+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:12.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[30436]: audit 2023-12-16T07:30:11.639046+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:12.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[30436]: cephadm 2023-12-16T07:30:11.640804+0000 mgr.y (mgr.14152) 42 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2023-12-16T07:30:12.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[30436]: audit 2023-12-16T07:30:11.641211+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:30:12.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[30436]: audit 2023-12-16T07:30:11.642402+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:30:12.933 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[30436]: audit 2023-12-16T07:30:11.643563+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:12.934 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[30436]: cephadm 2023-12-16T07:30:11.644927+0000 mgr.y (mgr.14152) 43 : cephadm [INF] Reconfiguring daemon mon.a on smithi007 2023-12-16T07:30:12.934 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:12 smithi007 bash[30436]: audit 2023-12-16T07:30:12.434328+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:12.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:12 smithi169 bash[25083]: cluster 2023-12-16T07:30:11.325439+0000 mgr.y (mgr.14152) 39 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:12.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:12 smithi169 bash[25083]: cephadm 2023-12-16T07:30:11.483525+0000 mgr.y (mgr.14152) 40 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:30:12.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:12 smithi169 bash[25083]: cephadm 2023-12-16T07:30:11.487775+0000 mgr.y (mgr.14152) 41 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:30:12.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:12 smithi169 bash[25083]: audit 2023-12-16T07:30:11.625332+0000 mon.a (mon.0) 172 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:12.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:12 smithi169 bash[25083]: audit 2023-12-16T07:30:11.633956+0000 mon.a (mon.0) 173 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:12.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:12 smithi169 bash[25083]: audit 2023-12-16T07:30:11.639046+0000 mon.a (mon.0) 174 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:12.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:12 smithi169 bash[25083]: cephadm 2023-12-16T07:30:11.640804+0000 mgr.y (mgr.14152) 42 : cephadm [INF] Reconfiguring mon.a (unknown last config time)... 2023-12-16T07:30:12.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:12 smithi169 bash[25083]: audit 2023-12-16T07:30:11.641211+0000 mon.a (mon.0) 175 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:30:12.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:12 smithi169 bash[25083]: audit 2023-12-16T07:30:11.642402+0000 mon.a (mon.0) 176 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:30:12.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:12 smithi169 bash[25083]: audit 2023-12-16T07:30:11.643563+0000 mon.a (mon.0) 177 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:12.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:12 smithi169 bash[25083]: cephadm 2023-12-16T07:30:11.644927+0000 mgr.y (mgr.14152) 43 : cephadm [INF] Reconfiguring daemon mon.a on smithi007 2023-12-16T07:30:12.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:12 smithi169 bash[25083]: audit 2023-12-16T07:30:12.434328+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:30:13.028 INFO:teuthology.orchestra.run.smithi169.stdout: 2023-12-16T07:30:13.028 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":3,"fsid":"8b71d3dc-9be4-11ee-95a5-87774f69a715","modified":"2023-12-16T07:30:06.434367Z","created":"2023-12-16T07:28:06.053524Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3301","nonce":0},{"type":"v1","addr":"172.21.15.7:6790","nonce":0}]},"addr":"172.21.15.7:6790/0","public_addr":"172.21.15.7:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:3300","nonce":0},{"type":"v1","addr":"172.21.15.169:6789","nonce":0}]},"addr":"172.21.15.169:6789/0","public_addr":"172.21.15.169:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2023-12-16T07:30:13.031 INFO:teuthology.orchestra.run.smithi169.stderr:dumped monmap epoch 3 2023-12-16T07:30:13.858 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-12-16T07:30:13.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph config generate-minimal-conf 2023-12-16T07:30:13.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:13 smithi169 bash[25083]: audit 2023-12-16T07:30:13.026791+0000 mon.a (mon.0) 179 : audit [DBG] from='client.? 172.21.15.169:0/3177614156' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T07:30:13.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:13 smithi007 bash[25009]: audit 2023-12-16T07:30:13.026791+0000 mon.a (mon.0) 179 : audit [DBG] from='client.? 172.21.15.169:0/3177614156' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T07:30:13.967 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:13 smithi007 bash[30436]: audit 2023-12-16T07:30:13.026791+0000 mon.a (mon.0) 179 : audit [DBG] from='client.? 172.21.15.169:0/3177614156' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-16T07:30:15.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:14 smithi007 bash[25009]: cluster 2023-12-16T07:30:13.325928+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:15.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:14 smithi007 bash[25009]: audit 2023-12-16T07:30:13.735221+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:15.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:14 smithi007 bash[25009]: cephadm 2023-12-16T07:30:13.736393+0000 mgr.y (mgr.14152) 45 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2023-12-16T07:30:15.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:14 smithi007 bash[25009]: audit 2023-12-16T07:30:13.736909+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:30:15.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:14 smithi007 bash[25009]: audit 2023-12-16T07:30:13.738420+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:30:15.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:14 smithi007 bash[25009]: audit 2023-12-16T07:30:13.739937+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:15.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:14 smithi007 bash[25009]: cephadm 2023-12-16T07:30:13.741601+0000 mgr.y (mgr.14152) 46 : cephadm [INF] Reconfiguring daemon mon.c on smithi007 2023-12-16T07:30:15.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:14 smithi007 bash[30436]: cluster 2023-12-16T07:30:13.325928+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:15.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:14 smithi007 bash[30436]: audit 2023-12-16T07:30:13.735221+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:15.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:14 smithi007 bash[30436]: cephadm 2023-12-16T07:30:13.736393+0000 mgr.y (mgr.14152) 45 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2023-12-16T07:30:15.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:14 smithi007 bash[30436]: audit 2023-12-16T07:30:13.736909+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:30:15.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:14 smithi007 bash[30436]: audit 2023-12-16T07:30:13.738420+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:30:15.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:14 smithi007 bash[30436]: audit 2023-12-16T07:30:13.739937+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:15.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:14 smithi007 bash[30436]: cephadm 2023-12-16T07:30:13.741601+0000 mgr.y (mgr.14152) 46 : cephadm [INF] Reconfiguring daemon mon.c on smithi007 2023-12-16T07:30:15.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:14 smithi169 bash[25083]: cluster 2023-12-16T07:30:13.325928+0000 mgr.y (mgr.14152) 44 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:15.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:14 smithi169 bash[25083]: audit 2023-12-16T07:30:13.735221+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:15.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:14 smithi169 bash[25083]: cephadm 2023-12-16T07:30:13.736393+0000 mgr.y (mgr.14152) 45 : cephadm [INF] Reconfiguring mon.c (monmap changed)... 2023-12-16T07:30:15.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:14 smithi169 bash[25083]: audit 2023-12-16T07:30:13.736909+0000 mon.a (mon.0) 181 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:30:15.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:14 smithi169 bash[25083]: audit 2023-12-16T07:30:13.738420+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:30:15.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:14 smithi169 bash[25083]: audit 2023-12-16T07:30:13.739937+0000 mon.a (mon.0) 183 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:15.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:14 smithi169 bash[25083]: cephadm 2023-12-16T07:30:13.741601+0000 mgr.y (mgr.14152) 46 : cephadm [INF] Reconfiguring daemon mon.c on smithi007 2023-12-16T07:30:16.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:15 smithi007 bash[25009]: audit 2023-12-16T07:30:15.385354+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:30:16.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:15 smithi007 bash[25009]: audit 2023-12-16T07:30:15.387332+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:30:16.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:15 smithi007 bash[30436]: audit 2023-12-16T07:30:15.385354+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:30:16.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:15 smithi007 bash[30436]: audit 2023-12-16T07:30:15.387332+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:30:16.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:15 smithi169 bash[25083]: audit 2023-12-16T07:30:15.385354+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:30:16.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:15 smithi169 bash[25083]: audit 2023-12-16T07:30:15.387332+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:30:16.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:16 smithi007 bash[25009]: cluster 2023-12-16T07:30:15.326421+0000 mgr.y (mgr.14152) 47 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:16.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:16 smithi007 bash[25009]: audit 2023-12-16T07:30:16.543813+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:16.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:16 smithi007 bash[25009]: audit 2023-12-16T07:30:16.545042+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:30:16.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:16 smithi007 bash[25009]: audit 2023-12-16T07:30:16.545981+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:30:16.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:16 smithi007 bash[25009]: audit 2023-12-16T07:30:16.546813+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:16.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:16 smithi007 bash[30436]: cluster 2023-12-16T07:30:15.326421+0000 mgr.y (mgr.14152) 47 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:16.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:16 smithi007 bash[30436]: audit 2023-12-16T07:30:16.543813+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:16.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:16 smithi007 bash[30436]: audit 2023-12-16T07:30:16.545042+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:30:16.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:16 smithi007 bash[30436]: audit 2023-12-16T07:30:16.545981+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:30:16.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:16 smithi007 bash[30436]: audit 2023-12-16T07:30:16.546813+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:16.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:16 smithi169 bash[25083]: cluster 2023-12-16T07:30:15.326421+0000 mgr.y (mgr.14152) 47 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:16.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:16 smithi169 bash[25083]: audit 2023-12-16T07:30:16.543813+0000 mon.a (mon.0) 186 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:16.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:16 smithi169 bash[25083]: audit 2023-12-16T07:30:16.545042+0000 mon.a (mon.0) 187 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:30:16.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:16 smithi169 bash[25083]: audit 2023-12-16T07:30:16.545981+0000 mon.a (mon.0) 188 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:30:16.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:16 smithi169 bash[25083]: audit 2023-12-16T07:30:16.546813+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:17.918 INFO:teuthology.orchestra.run.smithi007.stdout:# minimal ceph.conf for 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:30:17.918 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2023-12-16T07:30:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:30:17.919 INFO:teuthology.orchestra.run.smithi007.stdout: mon_host = [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2023-12-16T07:30:18.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:17 smithi169 bash[25083]: cephadm 2023-12-16T07:30:16.544687+0000 mgr.y (mgr.14152) 48 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2023-12-16T07:30:18.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:17 smithi169 bash[25083]: cephadm 2023-12-16T07:30:16.547859+0000 mgr.y (mgr.14152) 49 : cephadm [INF] Reconfiguring daemon mon.b on smithi169 2023-12-16T07:30:18.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:17 smithi169 bash[25083]: audit 2023-12-16T07:30:17.917208+0000 mon.a (mon.0) 190 : audit [DBG] from='client.? 172.21.15.7:0/2861462053' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:18.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:17 smithi007 bash[25009]: cephadm 2023-12-16T07:30:16.544687+0000 mgr.y (mgr.14152) 48 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2023-12-16T07:30:18.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[25009]: cephadm 2023-12-16T07:30:16.547859+0000 mgr.y (mgr.14152) 49 : cephadm [INF] Reconfiguring daemon mon.b on smithi169 2023-12-16T07:30:18.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[25009]: audit 2023-12-16T07:30:17.917208+0000 mon.a (mon.0) 190 : audit [DBG] from='client.? 172.21.15.7:0/2861462053' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:18.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[30436]: cephadm 2023-12-16T07:30:16.544687+0000 mgr.y (mgr.14152) 48 : cephadm [INF] Reconfiguring mon.b (monmap changed)... 2023-12-16T07:30:18.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[30436]: cephadm 2023-12-16T07:30:16.547859+0000 mgr.y (mgr.14152) 49 : cephadm [INF] Reconfiguring daemon mon.b on smithi169 2023-12-16T07:30:18.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[30436]: audit 2023-12-16T07:30:17.917208+0000 mon.a (mon.0) 190 : audit [DBG] from='client.? 172.21.15.7:0/2861462053' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:18.630 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-12-16T07:30:18.630 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-12-16T07:30:18.630 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.conf 2023-12-16T07:30:18.641 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-12-16T07:30:18.641 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:30:18.655 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T07:30:18.655 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.conf 2023-12-16T07:30:18.671 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T07:30:18.672 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:30:18.728 INFO:tasks.cephadm:Adding mgr.y on smithi007 2023-12-16T07:30:18.728 INFO:tasks.cephadm:Adding mgr.x on smithi169 2023-12-16T07:30:18.728 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch apply mgr '2;smithi007=y;smithi169=x' 2023-12-16T07:30:19.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:18 smithi169 bash[25083]: cluster 2023-12-16T07:30:17.326877+0000 mgr.y (mgr.14152) 50 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:19.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:18 smithi169 bash[25083]: audit 2023-12-16T07:30:18.396509+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:19.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:18 smithi169 bash[25083]: audit 2023-12-16T07:30:18.398840+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:19.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:18 smithi169 bash[25083]: audit 2023-12-16T07:30:18.400639+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:19.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:18 smithi169 bash[25083]: audit 2023-12-16T07:30:18.401881+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:19.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:18 smithi169 bash[25083]: audit 2023-12-16T07:30:18.542200+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:19.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:18 smithi169 bash[25083]: audit 2023-12-16T07:30:18.550858+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:19.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:18 smithi169 bash[25083]: audit 2023-12-16T07:30:18.556072+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:19.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[25009]: cluster 2023-12-16T07:30:17.326877+0000 mgr.y (mgr.14152) 50 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:19.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[25009]: audit 2023-12-16T07:30:18.396509+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:19.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[25009]: audit 2023-12-16T07:30:18.398840+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:19.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[25009]: audit 2023-12-16T07:30:18.400639+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:19.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[25009]: audit 2023-12-16T07:30:18.401881+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:19.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[25009]: audit 2023-12-16T07:30:18.542200+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:19.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[25009]: audit 2023-12-16T07:30:18.550858+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:19.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[25009]: audit 2023-12-16T07:30:18.556072+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:19.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[30436]: cluster 2023-12-16T07:30:17.326877+0000 mgr.y (mgr.14152) 50 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:19.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[30436]: audit 2023-12-16T07:30:18.396509+0000 mon.a (mon.0) 191 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:19.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[30436]: audit 2023-12-16T07:30:18.398840+0000 mon.a (mon.0) 192 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:19.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[30436]: audit 2023-12-16T07:30:18.400639+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:19.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[30436]: audit 2023-12-16T07:30:18.401881+0000 mon.a (mon.0) 194 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:19.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[30436]: audit 2023-12-16T07:30:18.542200+0000 mon.a (mon.0) 195 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:19.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[30436]: audit 2023-12-16T07:30:18.550858+0000 mon.a (mon.0) 196 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:19.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:18 smithi007 bash[30436]: audit 2023-12-16T07:30:18.556072+0000 mon.a (mon.0) 197 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:20.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:19 smithi007 bash[25009]: cephadm 2023-12-16T07:30:18.403674+0000 mgr.y (mgr.14152) 51 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:30:20.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:19 smithi007 bash[25009]: cephadm 2023-12-16T07:30:18.404244+0000 mgr.y (mgr.14152) 52 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:30:20.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:19 smithi007 bash[30436]: cephadm 2023-12-16T07:30:18.403674+0000 mgr.y (mgr.14152) 51 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:30:20.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:19 smithi007 bash[30436]: cephadm 2023-12-16T07:30:18.404244+0000 mgr.y (mgr.14152) 52 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:30:20.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:19 smithi169 bash[25083]: cephadm 2023-12-16T07:30:18.403674+0000 mgr.y (mgr.14152) 51 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:30:20.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:19 smithi169 bash[25083]: cephadm 2023-12-16T07:30:18.404244+0000 mgr.y (mgr.14152) 52 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:30:21.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:20 smithi007 bash[25009]: cluster 2023-12-16T07:30:19.327250+0000 mgr.y (mgr.14152) 53 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:21.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:20 smithi007 bash[30436]: cluster 2023-12-16T07:30:19.327250+0000 mgr.y (mgr.14152) 53 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:21.444 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:20 smithi169 bash[25083]: cluster 2023-12-16T07:30:19.327250+0000 mgr.y (mgr.14152) 53 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:21.590 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled mgr update... 2023-12-16T07:30:22.364 DEBUG:teuthology.orchestra.run.smithi169:mgr.x> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.x.service 2023-12-16T07:30:22.371 INFO:tasks.cephadm:Deploying OSDs... 2023-12-16T07:30:22.371 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-12-16T07:30:22.371 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2023-12-16T07:30:22.380 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-16T07:30:22.380 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2023-12-16T07:30:22.388 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-16T07:30:22.389 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T07:30:22.389 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 360 Links: 1 2023-12-16T07:30:22.389 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T07:30:22.389 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-12-16 07:25:37.394169703 +0000 2023-12-16T07:30:22.389 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-12-16 07:25:37.114176146 +0000 2023-12-16T07:30:22.389 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-12-16 07:25:37.114176146 +0000 2023-12-16T07:30:22.389 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2023-12-16T07:30:22.389 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-16T07:30:22.444 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-12-16T07:30:22.444 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-12-16T07:30:22.444 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000322553 s, 1.6 MB/s 2023-12-16T07:30:22.445 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-16T07:30:22.456 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2023-12-16T07:30:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-16T07:30:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T07:30:22.466 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 367 Links: 1 2023-12-16T07:30:22.466 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T07:30:22.466 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-12-16 07:25:37.930157368 +0000 2023-12-16T07:30:22.466 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-12-16 07:25:37.634164179 +0000 2023-12-16T07:30:22.466 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-12-16 07:25:37.634164179 +0000 2023-12-16T07:30:22.466 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2023-12-16T07:30:22.467 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-16T07:30:22.482 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-12-16T07:30:22.482 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-12-16T07:30:22.482 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000311987 s, 1.6 MB/s 2023-12-16T07:30:22.483 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-16T07:30:22.538 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2023-12-16T07:30:22.546 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-16T07:30:22.546 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T07:30:22.547 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 372 Links: 1 2023-12-16T07:30:22.547 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T07:30:22.547 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-12-16 07:25:38.458145217 +0000 2023-12-16T07:30:22.547 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-12-16 07:25:38.154152214 +0000 2023-12-16T07:30:22.547 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-12-16 07:25:38.154152214 +0000 2023-12-16T07:30:22.547 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2023-12-16T07:30:22.547 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-16T07:30:22.601 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-12-16T07:30:22.601 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-12-16T07:30:22.602 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000182043 s, 2.8 MB/s 2023-12-16T07:30:22.606 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-16T07:30:22.614 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2023-12-16T07:30:22.621 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-16T07:30:22.621 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T07:30:22.621 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 6h/6d Inode: 377 Links: 1 2023-12-16T07:30:22.621 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T07:30:22.621 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-12-16 07:25:38.998132790 +0000 2023-12-16T07:30:22.621 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-12-16 07:25:38.702139603 +0000 2023-12-16T07:30:22.621 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-12-16 07:25:38.702139603 +0000 2023-12-16T07:30:22.622 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: - 2023-12-16T07:30:22.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-16T07:30:22.677 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-12-16T07:30:22.677 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-12-16T07:30:22.678 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000299724 s, 1.7 MB/s 2023-12-16T07:30:22.678 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-16T07:30:22.687 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T07:30:22.688 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2023-12-16T07:30:22.695 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-16T07:30:22.695 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2023-12-16T07:30:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-16T07:30:22.703 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T07:30:22.703 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 6h/6d Inode: 360 Links: 1 2023-12-16T07:30:22.703 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T07:30:22.703 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2023-12-16 07:25:34.059792967 +0000 2023-12-16T07:30:22.703 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2023-12-16 07:25:33.747800158 +0000 2023-12-16T07:30:22.703 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2023-12-16 07:25:33.747800158 +0000 2023-12-16T07:30:22.704 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2023-12-16T07:30:22.704 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-16T07:30:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2023-12-16T07:30:22.714 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2023-12-16T07:30:22.715 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000402381 s, 1.3 MB/s 2023-12-16T07:30:22.715 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-16T07:30:22.723 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2023-12-16T07:30:22.771 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-16T07:30:22.771 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T07:30:22.772 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 6h/6d Inode: 365 Links: 1 2023-12-16T07:30:22.772 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T07:30:22.772 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2023-12-16 07:25:34.591780704 +0000 2023-12-16T07:30:22.772 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2023-12-16 07:25:34.267788171 +0000 2023-12-16T07:30:22.772 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2023-12-16 07:25:34.267788171 +0000 2023-12-16T07:30:22.772 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2023-12-16T07:30:22.772 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-16T07:30:22.784 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2023-12-16T07:30:22.785 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2023-12-16T07:30:22.785 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000372169 s, 1.4 MB/s 2023-12-16T07:30:22.786 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-16T07:30:22.795 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2023-12-16T07:30:22.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[25009]: cluster 2023-12-16T07:30:21.327662+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:22.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[25009]: audit 2023-12-16T07:30:21.577925+0000 mgr.y (mgr.14152) 55 : audit [DBG] from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi169=x", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:30:22.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[25009]: cephadm 2023-12-16T07:30:21.581456+0000 mgr.y (mgr.14152) 56 : cephadm [INF] Saving service mgr spec with placement smithi007=y;smithi169=x;count:2 2023-12-16T07:30:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[25009]: audit 2023-12-16T07:30:21.588034+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[25009]: audit 2023-12-16T07:30:21.632668+0000 mon.a (mon.0) 199 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[25009]: audit 2023-12-16T07:30:21.635555+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[25009]: audit 2023-12-16T07:30:21.637570+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[25009]: audit 2023-12-16T07:30:21.922692+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[25009]: audit 2023-12-16T07:30:21.924531+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:30:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[25009]: audit 2023-12-16T07:30:21.928244+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-12-16T07:30:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[25009]: audit 2023-12-16T07:30:21.929324+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:30:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[25009]: audit 2023-12-16T07:30:21.930513+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[25009]: cephadm 2023-12-16T07:30:21.931726+0000 mgr.y (mgr.14152) 57 : cephadm [INF] Deploying daemon mgr.x on smithi169 2023-12-16T07:30:22.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[30436]: cluster 2023-12-16T07:30:21.327662+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[30436]: audit 2023-12-16T07:30:21.577925+0000 mgr.y (mgr.14152) 55 : audit [DBG] from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi169=x", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:30:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[30436]: cephadm 2023-12-16T07:30:21.581456+0000 mgr.y (mgr.14152) 56 : cephadm [INF] Saving service mgr spec with placement smithi007=y;smithi169=x;count:2 2023-12-16T07:30:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[30436]: audit 2023-12-16T07:30:21.588034+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[30436]: audit 2023-12-16T07:30:21.632668+0000 mon.a (mon.0) 199 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[30436]: audit 2023-12-16T07:30:21.635555+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[30436]: audit 2023-12-16T07:30:21.637570+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[30436]: audit 2023-12-16T07:30:21.922692+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[30436]: audit 2023-12-16T07:30:21.924531+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:30:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[30436]: audit 2023-12-16T07:30:21.928244+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-12-16T07:30:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[30436]: audit 2023-12-16T07:30:21.929324+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:30:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[30436]: audit 2023-12-16T07:30:21.930513+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:22 smithi007 bash[30436]: cephadm 2023-12-16T07:30:21.931726+0000 mgr.y (mgr.14152) 57 : cephadm [INF] Deploying daemon mgr.x on smithi169 2023-12-16T07:30:22.843 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-16T07:30:22.844 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T07:30:22.844 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 6h/6d Inode: 373 Links: 1 2023-12-16T07:30:22.844 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T07:30:22.844 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2023-12-16 07:25:35.087769270 +0000 2023-12-16T07:30:22.844 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2023-12-16 07:25:34.787776185 +0000 2023-12-16T07:30:22.844 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2023-12-16 07:25:34.787776185 +0000 2023-12-16T07:30:22.844 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2023-12-16T07:30:22.844 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-16T07:30:22.855 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:22 smithi169 bash[25083]: cluster 2023-12-16T07:30:21.327662+0000 mgr.y (mgr.14152) 54 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:22.855 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:22 smithi169 bash[25083]: audit 2023-12-16T07:30:21.577925+0000 mgr.y (mgr.14152) 55 : audit [DBG] from='client.24107 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi169=x", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:30:22.855 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:22 smithi169 bash[25083]: cephadm 2023-12-16T07:30:21.581456+0000 mgr.y (mgr.14152) 56 : cephadm [INF] Saving service mgr spec with placement smithi007=y;smithi169=x;count:2 2023-12-16T07:30:22.855 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:22 smithi169 bash[25083]: audit 2023-12-16T07:30:21.588034+0000 mon.a (mon.0) 198 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:22.855 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:22 smithi169 bash[25083]: audit 2023-12-16T07:30:21.632668+0000 mon.a (mon.0) 199 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:22.855 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:22 smithi169 bash[25083]: audit 2023-12-16T07:30:21.635555+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:22.855 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:22 smithi169 bash[25083]: audit 2023-12-16T07:30:21.637570+0000 mon.a (mon.0) 201 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:22.856 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:22 smithi169 bash[25083]: audit 2023-12-16T07:30:21.922692+0000 mon.a (mon.0) 202 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:22.856 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:22 smithi169 bash[25083]: audit 2023-12-16T07:30:21.924531+0000 mon.a (mon.0) 203 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:30:22.856 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:22 smithi169 bash[25083]: audit 2023-12-16T07:30:21.928244+0000 mon.a (mon.0) 204 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-12-16T07:30:22.856 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:22 smithi169 bash[25083]: audit 2023-12-16T07:30:21.929324+0000 mon.a (mon.0) 205 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:30:22.856 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:22 smithi169 bash[25083]: audit 2023-12-16T07:30:21.930513+0000 mon.a (mon.0) 206 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:22.856 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:22 smithi169 bash[25083]: cephadm 2023-12-16T07:30:21.931726+0000 mgr.y (mgr.14152) 57 : cephadm [INF] Deploying daemon mgr.x on smithi169 2023-12-16T07:30:22.857 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2023-12-16T07:30:22.857 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2023-12-16T07:30:22.857 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000443283 s, 1.2 MB/s 2023-12-16T07:30:22.858 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-16T07:30:22.867 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2023-12-16T07:30:22.872 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-16T07:30:22.872 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-16T07:30:22.872 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 6h/6d Inode: 378 Links: 1 2023-12-16T07:30:22.872 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-16T07:30:22.872 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2023-12-16 07:25:35.591757653 +0000 2023-12-16T07:30:22.872 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2023-12-16 07:25:35.299764384 +0000 2023-12-16T07:30:22.872 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2023-12-16 07:25:35.299764384 +0000 2023-12-16T07:30:22.872 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: - 2023-12-16T07:30:22.872 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-16T07:30:22.927 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2023-12-16T07:30:22.927 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2023-12-16T07:30:22.927 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000416314 s, 1.2 MB/s 2023-12-16T07:30:22.928 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-16T07:30:22.937 INFO:tasks.cephadm:Deploying osd.0 on smithi007 with /dev/vg_nvme/lv_4... 2023-12-16T07:30:22.937 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-12-16T07:30:23.244 INFO:journalctl@ceph.mgr.x.smithi169.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:30:24.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:24 smithi007 bash[25009]: cluster 2023-12-16T07:30:23.328220+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:24.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:24 smithi007 bash[30436]: cluster 2023-12-16T07:30:23.328220+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:24.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:24 smithi169 bash[25083]: cluster 2023-12-16T07:30:23.328220+0000 mgr.y (mgr.14152) 58 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:25.796 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:25 smithi169 bash[25855]: debug 2023-12-16T07:30:25.630+0000 7fa696b8f700 1 -- 172.21.15.169:0/4087709148 <== mon.2 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5627b0838340 con 0x5627b15b4400 2023-12-16T07:30:25.797 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:25 smithi169 bash[25855]: debug 2023-12-16T07:30:25.790+0000 7fa69f7fe000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T07:30:25.797 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:25 smithi169 bash[25083]: audit 2023-12-16T07:30:24.639187+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:25.797 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:25 smithi169 bash[25083]: audit 2023-12-16T07:30:24.643250+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:25.797 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:25 smithi169 bash[25083]: audit 2023-12-16T07:30:24.645823+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:25.797 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:25 smithi169 bash[25083]: audit 2023-12-16T07:30:24.647577+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:25.918 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:25 smithi007 bash[30436]: audit 2023-12-16T07:30:24.639187+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:25.918 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:25 smithi007 bash[30436]: audit 2023-12-16T07:30:24.643250+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:25.918 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:25 smithi007 bash[30436]: audit 2023-12-16T07:30:24.645823+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:25.918 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:25 smithi007 bash[30436]: audit 2023-12-16T07:30:24.647577+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:25.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:25 smithi007 bash[25009]: audit 2023-12-16T07:30:24.639187+0000 mon.a (mon.0) 207 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:25.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:25 smithi007 bash[25009]: audit 2023-12-16T07:30:24.643250+0000 mon.a (mon.0) 208 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:25.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:25 smithi007 bash[25009]: audit 2023-12-16T07:30:24.645823+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:25.918 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:25 smithi007 bash[25009]: audit 2023-12-16T07:30:24.647577+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:26.445 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-12-16T07:30:26.445 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-12-16T07:30:26.445 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2023-12-16T07:30:26.445 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2023-12-16T07:30:26.445 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01652 s, 635 MB/s 2023-12-16T07:30:26.445 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2023-12-16T07:30:26.741 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:26 smithi169 bash[25083]: cluster 2023-12-16T07:30:25.328692+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:26.741 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:26 smithi169 bash[25083]: audit 2023-12-16T07:30:26.497557+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:26.742 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:26 smithi169 bash[25855]: debug 2023-12-16T07:30:26.462+0000 7fa69f7fe000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T07:30:26.742 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:26 smithi169 bash[25855]: debug 2023-12-16T07:30:26.582+0000 7fa69f7fe000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T07:30:26.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:26 smithi007 bash[25009]: cluster 2023-12-16T07:30:25.328692+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:26.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:26 smithi007 bash[25009]: audit 2023-12-16T07:30:26.497557+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:26.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:26 smithi007 bash[30436]: cluster 2023-12-16T07:30:25.328692+0000 mgr.y (mgr.14152) 59 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:26.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:26 smithi007 bash[30436]: audit 2023-12-16T07:30:26.497557+0000 mon.a (mon.0) 211 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:27.066 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:26 smithi169 bash[25855]: debug 2023-12-16T07:30:26.738+0000 7fa69f7fe000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:30:27.192 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi007:vg_nvme/lv_4 2023-12-16T07:30:27.357 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:27 smithi169 bash[25855]: debug 2023-12-16T07:30:27.066+0000 7fa69f7fe000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T07:30:27.357 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:27 smithi169 bash[25855]: debug 2023-12-16T07:30:27.158+0000 7fa69f7fe000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T07:30:27.667 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:27 smithi169 bash[25855]: debug 2023-12-16T07:30:27.354+0000 7fa69f7fe000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T07:30:28.555 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:28 smithi169 bash[25855]: debug 2023-12-16T07:30:28.246+0000 7fa69f7fe000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T07:30:28.555 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:28 smithi169 bash[25855]: debug 2023-12-16T07:30:28.350+0000 7fa69f7fe000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T07:30:28.899 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:28 smithi169 bash[25855]: debug 2023-12-16T07:30:28.554+0000 7fa69f7fe000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T07:30:28.899 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:28 smithi169 bash[25083]: cluster 2023-12-16T07:30:27.329207+0000 mgr.y (mgr.14152) 60 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:28.899 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:28 smithi169 bash[25083]: audit 2023-12-16T07:30:27.681467+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:28.899 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:28 smithi169 bash[25083]: audit 2023-12-16T07:30:27.689041+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:28.900 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:28 smithi169 bash[25083]: cephadm 2023-12-16T07:30:27.692682+0000 mgr.y (mgr.14152) 61 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2023-12-16T07:30:28.900 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:28 smithi169 bash[25083]: audit 2023-12-16T07:30:27.693308+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:30:28.900 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:28 smithi169 bash[25083]: audit 2023-12-16T07:30:27.694950+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:30:28.900 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:28 smithi169 bash[25083]: audit 2023-12-16T07:30:27.696216+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:28.900 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:28 smithi169 bash[25083]: cephadm 2023-12-16T07:30:27.697462+0000 mgr.y (mgr.14152) 62 : cephadm [INF] Reconfiguring daemon mgr.y on smithi007 2023-12-16T07:30:29.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[25009]: cluster 2023-12-16T07:30:27.329207+0000 mgr.y (mgr.14152) 60 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:29.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[25009]: audit 2023-12-16T07:30:27.681467+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:29.053 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[25009]: audit 2023-12-16T07:30:27.689041+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:29.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[25009]: cephadm 2023-12-16T07:30:27.692682+0000 mgr.y (mgr.14152) 61 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2023-12-16T07:30:29.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[25009]: audit 2023-12-16T07:30:27.693308+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:30:29.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[25009]: audit 2023-12-16T07:30:27.694950+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:30:29.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[25009]: audit 2023-12-16T07:30:27.696216+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:29.054 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[25009]: cephadm 2023-12-16T07:30:27.697462+0000 mgr.y (mgr.14152) 62 : cephadm [INF] Reconfiguring daemon mgr.y on smithi007 2023-12-16T07:30:29.054 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[30436]: cluster 2023-12-16T07:30:27.329207+0000 mgr.y (mgr.14152) 60 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:29.054 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[30436]: audit 2023-12-16T07:30:27.681467+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:29.054 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[30436]: audit 2023-12-16T07:30:27.689041+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:29.054 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[30436]: cephadm 2023-12-16T07:30:27.692682+0000 mgr.y (mgr.14152) 61 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2023-12-16T07:30:29.054 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[30436]: audit 2023-12-16T07:30:27.693308+0000 mon.a (mon.0) 214 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:30:29.054 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[30436]: audit 2023-12-16T07:30:27.694950+0000 mon.a (mon.0) 215 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:30:29.054 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[30436]: audit 2023-12-16T07:30:27.696216+0000 mon.a (mon.0) 216 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:29.054 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:28 smithi007 bash[30436]: cephadm 2023-12-16T07:30:27.697462+0000 mgr.y (mgr.14152) 62 : cephadm [INF] Reconfiguring daemon mgr.y on smithi007 2023-12-16T07:30:29.182 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:28 smithi169 bash[25855]: debug 2023-12-16T07:30:28.898+0000 7fa69f7fe000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T07:30:29.591 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:29 smithi169 bash[25855]: debug 2023-12-16T07:30:29.226+0000 7fa69f7fe000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T07:30:29.857 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:29 smithi169 bash[25855]: debug 2023-12-16T07:30:29.590+0000 7fa69f7fe000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T07:30:30.182 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:29 smithi169 bash[25855]: debug 2023-12-16T07:30:29.858+0000 7fa69f7fe000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:30:30.797 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:30 smithi169 bash[25083]: cluster 2023-12-16T07:30:29.329758+0000 mgr.y (mgr.14152) 63 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:30.798 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:30 smithi169 bash[25855]: debug 2023-12-16T07:30:30.538+0000 7fa69f7fe000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T07:30:30.798 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:30 smithi169 bash[25855]: debug 2023-12-16T07:30:30.638+0000 7fa69f7fe000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T07:30:31.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:30 smithi007 bash[25009]: cluster 2023-12-16T07:30:29.329758+0000 mgr.y (mgr.14152) 63 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:31.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:30 smithi007 bash[30436]: cluster 2023-12-16T07:30:29.329758+0000 mgr.y (mgr.14152) 63 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:31.182 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:30 smithi169 bash[25855]: debug 2023-12-16T07:30:30.798+0000 7fa69f7fe000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T07:30:31.182 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:30 smithi169 bash[25855]: debug 2023-12-16T07:30:30.902+0000 7fa69f7fe000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T07:30:31.182 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:31 smithi169 bash[25855]: debug 2023-12-16T07:30:31.038+0000 7fa69f7fe000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T07:30:31.628 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:31 smithi169 bash[25855]: debug 2023-12-16T07:30:31.330+0000 7fa69f7fe000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T07:30:31.629 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:31 smithi169 bash[25855]: debug 2023-12-16T07:30:31.450+0000 7fa69f7fe000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T07:30:31.932 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:31 smithi169 bash[25855]: debug 2023-12-16T07:30:31.626+0000 7fa69f7fe000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T07:30:31.932 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:31 smithi169 bash[25855]: debug 2023-12-16T07:30:31.746+0000 7fa69f7fe000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T07:30:31.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:31 smithi169 bash[25083]: audit 2023-12-16T07:30:30.867100+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:31.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:31 smithi169 bash[25083]: audit 2023-12-16T07:30:30.869469+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:31.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:31 smithi169 bash[25083]: audit 2023-12-16T07:30:30.871425+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:31.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:31 smithi169 bash[25083]: audit 2023-12-16T07:30:30.872718+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:31.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:31 smithi169 bash[25083]: audit 2023-12-16T07:30:30.880927+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:31.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:31 smithi169 bash[25083]: audit 2023-12-16T07:30:31.576903+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:30:31.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:31 smithi169 bash[25083]: audit 2023-12-16T07:30:31.583787+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:30:31.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:31 smithi169 bash[25083]: audit 2023-12-16T07:30:31.585338+0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:32.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[25009]: audit 2023-12-16T07:30:30.867100+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:32.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[25009]: audit 2023-12-16T07:30:30.869469+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:32.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[25009]: audit 2023-12-16T07:30:30.871425+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:32.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[25009]: audit 2023-12-16T07:30:30.872718+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:32.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[25009]: audit 2023-12-16T07:30:30.880927+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:32.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[25009]: audit 2023-12-16T07:30:31.576903+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:30:32.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[25009]: audit 2023-12-16T07:30:31.583787+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:30:32.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[25009]: audit 2023-12-16T07:30:31.585338+0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:32.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[30436]: audit 2023-12-16T07:30:30.867100+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:32.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[30436]: audit 2023-12-16T07:30:30.869469+0000 mon.a (mon.0) 218 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:32.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[30436]: audit 2023-12-16T07:30:30.871425+0000 mon.a (mon.0) 219 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:32.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[30436]: audit 2023-12-16T07:30:30.872718+0000 mon.a (mon.0) 220 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:32.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[30436]: audit 2023-12-16T07:30:30.880927+0000 mon.a (mon.0) 221 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:32.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[30436]: audit 2023-12-16T07:30:31.576903+0000 mon.a (mon.0) 222 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:30:32.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[30436]: audit 2023-12-16T07:30:31.583787+0000 mon.a (mon.0) 223 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:30:32.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:31 smithi007 bash[30436]: audit 2023-12-16T07:30:31.585338+0000 mon.a (mon.0) 224 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:32.932 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:30:32 smithi169 bash[25855]: debug 2023-12-16T07:30:32.586+0000 7fa69f7fe000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T07:30:33.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:32 smithi007 bash[25009]: cluster 2023-12-16T07:30:31.330272+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:33.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:32 smithi007 bash[25009]: audit 2023-12-16T07:30:31.570351+0000 mgr.y (mgr.14152) 65 : audit [DBG] from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:30:33.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:32 smithi007 bash[25009]: cluster 2023-12-16T07:30:32.607077+0000 mon.a (mon.0) 225 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:30:33.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:32 smithi007 bash[25009]: audit 2023-12-16T07:30:32.610974+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.? 172.21.15.169:0/1079037998' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:30:33.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:32 smithi007 bash[25009]: audit 2023-12-16T07:30:32.612202+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.? 172.21.15.169:0/1079037998' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:30:33.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:32 smithi007 bash[25009]: audit 2023-12-16T07:30:32.614578+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.? 172.21.15.169:0/1079037998' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:30:33.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:32 smithi007 bash[25009]: audit 2023-12-16T07:30:32.615675+0000 mon.a (mon.0) 229 : audit [DBG] from='mgr.? 172.21.15.169:0/1079037998' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:30:33.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:32 smithi007 bash[30436]: cluster 2023-12-16T07:30:31.330272+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:33.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:32 smithi007 bash[30436]: audit 2023-12-16T07:30:31.570351+0000 mgr.y (mgr.14152) 65 : audit [DBG] from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:30:33.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:32 smithi007 bash[30436]: cluster 2023-12-16T07:30:32.607077+0000 mon.a (mon.0) 225 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:30:33.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:32 smithi007 bash[30436]: audit 2023-12-16T07:30:32.610974+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.? 172.21.15.169:0/1079037998' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:30:33.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:32 smithi007 bash[30436]: audit 2023-12-16T07:30:32.612202+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.? 172.21.15.169:0/1079037998' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:30:33.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:32 smithi007 bash[30436]: audit 2023-12-16T07:30:32.614578+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.? 172.21.15.169:0/1079037998' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:30:33.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:32 smithi007 bash[30436]: audit 2023-12-16T07:30:32.615675+0000 mon.a (mon.0) 229 : audit [DBG] from='mgr.? 172.21.15.169:0/1079037998' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:30:33.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:32 smithi169 bash[25083]: cluster 2023-12-16T07:30:31.330272+0000 mgr.y (mgr.14152) 64 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:33.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:32 smithi169 bash[25083]: audit 2023-12-16T07:30:31.570351+0000 mgr.y (mgr.14152) 65 : audit [DBG] from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:30:33.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:32 smithi169 bash[25083]: cluster 2023-12-16T07:30:32.607077+0000 mon.a (mon.0) 225 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:30:33.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:32 smithi169 bash[25083]: audit 2023-12-16T07:30:32.610974+0000 mon.a (mon.0) 226 : audit [DBG] from='mgr.? 172.21.15.169:0/1079037998' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:30:33.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:32 smithi169 bash[25083]: audit 2023-12-16T07:30:32.612202+0000 mon.a (mon.0) 227 : audit [DBG] from='mgr.? 172.21.15.169:0/1079037998' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:30:33.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:32 smithi169 bash[25083]: audit 2023-12-16T07:30:32.614578+0000 mon.a (mon.0) 228 : audit [DBG] from='mgr.? 172.21.15.169:0/1079037998' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:30:33.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:32 smithi169 bash[25083]: audit 2023-12-16T07:30:32.615675+0000 mon.a (mon.0) 229 : audit [DBG] from='mgr.? 172.21.15.169:0/1079037998' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:30:34.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:33 smithi007 bash[30436]: cluster 2023-12-16T07:30:32.997865+0000 mon.a (mon.0) 230 : cluster [DBG] mgrmap e14: y(active, since 77s), standbys: x 2023-12-16T07:30:34.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:33 smithi007 bash[30436]: audit 2023-12-16T07:30:32.998062+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:30:34.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:33 smithi007 bash[25009]: cluster 2023-12-16T07:30:32.997865+0000 mon.a (mon.0) 230 : cluster [DBG] mgrmap e14: y(active, since 77s), standbys: x 2023-12-16T07:30:34.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:33 smithi007 bash[25009]: audit 2023-12-16T07:30:32.998062+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:30:34.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:33 smithi169 bash[25083]: cluster 2023-12-16T07:30:32.997865+0000 mon.a (mon.0) 230 : cluster [DBG] mgrmap e14: y(active, since 77s), standbys: x 2023-12-16T07:30:34.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:33 smithi169 bash[25083]: audit 2023-12-16T07:30:32.998062+0000 mon.a (mon.0) 231 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:30:35.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:35 smithi007 bash[25009]: cluster 2023-12-16T07:30:33.330765+0000 mgr.y (mgr.14152) 66 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:35.302 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:35 smithi007 bash[30436]: cluster 2023-12-16T07:30:33.330765+0000 mgr.y (mgr.14152) 66 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:35.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:35 smithi169 bash[25083]: cluster 2023-12-16T07:30:33.330765+0000 mgr.y (mgr.14152) 66 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:37.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:37 smithi007 bash[25009]: cluster 2023-12-16T07:30:35.331110+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:37.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:37 smithi007 bash[30436]: cluster 2023-12-16T07:30:35.331110+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:37.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:37 smithi169 bash[25083]: cluster 2023-12-16T07:30:35.331110+0000 mgr.y (mgr.14152) 67 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:38.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:38 smithi007 bash[25009]: audit 2023-12-16T07:30:37.772411+0000 mon.a (mon.0) 232 : audit [INF] from='client.? 172.21.15.7:0/1614950466' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d8aa3a96-fde6-48ee-9a0a-4ad9a0d4f427"}]: dispatch 2023-12-16T07:30:38.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:38 smithi007 bash[25009]: audit 2023-12-16T07:30:37.779515+0000 mon.a (mon.0) 233 : audit [INF] from='client.? 172.21.15.7:0/1614950466' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d8aa3a96-fde6-48ee-9a0a-4ad9a0d4f427"}]': finished 2023-12-16T07:30:38.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:38 smithi007 bash[25009]: cluster 2023-12-16T07:30:37.779641+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2023-12-16T07:30:38.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:38 smithi007 bash[25009]: audit 2023-12-16T07:30:37.779817+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:30:38.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:38 smithi007 bash[30436]: audit 2023-12-16T07:30:37.772411+0000 mon.a (mon.0) 232 : audit [INF] from='client.? 172.21.15.7:0/1614950466' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d8aa3a96-fde6-48ee-9a0a-4ad9a0d4f427"}]: dispatch 2023-12-16T07:30:38.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:38 smithi007 bash[30436]: audit 2023-12-16T07:30:37.779515+0000 mon.a (mon.0) 233 : audit [INF] from='client.? 172.21.15.7:0/1614950466' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d8aa3a96-fde6-48ee-9a0a-4ad9a0d4f427"}]': finished 2023-12-16T07:30:38.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:38 smithi007 bash[30436]: cluster 2023-12-16T07:30:37.779641+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2023-12-16T07:30:38.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:38 smithi007 bash[30436]: audit 2023-12-16T07:30:37.779817+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:30:38.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:38 smithi169 bash[25083]: audit 2023-12-16T07:30:37.772411+0000 mon.a (mon.0) 232 : audit [INF] from='client.? 172.21.15.7:0/1614950466' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d8aa3a96-fde6-48ee-9a0a-4ad9a0d4f427"}]: dispatch 2023-12-16T07:30:38.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:38 smithi169 bash[25083]: audit 2023-12-16T07:30:37.779515+0000 mon.a (mon.0) 233 : audit [INF] from='client.? 172.21.15.7:0/1614950466' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d8aa3a96-fde6-48ee-9a0a-4ad9a0d4f427"}]': finished 2023-12-16T07:30:38.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:38 smithi169 bash[25083]: cluster 2023-12-16T07:30:37.779641+0000 mon.a (mon.0) 234 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2023-12-16T07:30:38.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:38 smithi169 bash[25083]: audit 2023-12-16T07:30:37.779817+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:30:39.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:39 smithi007 bash[25009]: cluster 2023-12-16T07:30:37.331638+0000 mgr.y (mgr.14152) 68 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:39.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:39 smithi007 bash[25009]: audit 2023-12-16T07:30:38.493555+0000 mon.a (mon.0) 236 : audit [DBG] from='client.? 172.21.15.7:0/1534072442' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:30:39.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:39 smithi007 bash[30436]: cluster 2023-12-16T07:30:37.331638+0000 mgr.y (mgr.14152) 68 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:39.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:39 smithi007 bash[30436]: audit 2023-12-16T07:30:38.493555+0000 mon.a (mon.0) 236 : audit [DBG] from='client.? 172.21.15.7:0/1534072442' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:30:39.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:39 smithi169 bash[25083]: cluster 2023-12-16T07:30:37.331638+0000 mgr.y (mgr.14152) 68 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:39.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:39 smithi169 bash[25083]: audit 2023-12-16T07:30:38.493555+0000 mon.a (mon.0) 236 : audit [DBG] from='client.? 172.21.15.7:0/1534072442' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:30:41.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:41 smithi007 bash[25009]: cluster 2023-12-16T07:30:39.332206+0000 mgr.y (mgr.14152) 69 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:41.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:41 smithi007 bash[30436]: cluster 2023-12-16T07:30:39.332206+0000 mgr.y (mgr.14152) 69 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:41.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:41 smithi169 bash[25083]: cluster 2023-12-16T07:30:39.332206+0000 mgr.y (mgr.14152) 69 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:43.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:43 smithi007 bash[25009]: cluster 2023-12-16T07:30:41.332619+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:43.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:43 smithi007 bash[30436]: cluster 2023-12-16T07:30:41.332619+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:43.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:43 smithi169 bash[25083]: cluster 2023-12-16T07:30:41.332619+0000 mgr.y (mgr.14152) 70 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:44.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:44 smithi007 bash[30436]: cluster 2023-12-16T07:30:43.333031+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:44.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:44 smithi007 bash[25009]: cluster 2023-12-16T07:30:43.333031+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:44.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:44 smithi169 bash[25083]: cluster 2023-12-16T07:30:43.333031+0000 mgr.y (mgr.14152) 71 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:46.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:46 smithi169 bash[25083]: cluster 2023-12-16T07:30:45.333524+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:46.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:46 smithi007 bash[25009]: cluster 2023-12-16T07:30:45.333524+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:46.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:46 smithi007 bash[30436]: cluster 2023-12-16T07:30:45.333524+0000 mgr.y (mgr.14152) 72 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:49.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:48 smithi007 bash[25009]: cluster 2023-12-16T07:30:47.333955+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:49.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:48 smithi007 bash[25009]: audit 2023-12-16T07:30:48.280602+0000 mon.a (mon.0) 237 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-16T07:30:49.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:48 smithi007 bash[25009]: audit 2023-12-16T07:30:48.282757+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:49.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:48 smithi007 bash[30436]: cluster 2023-12-16T07:30:47.333955+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:49.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:48 smithi007 bash[30436]: audit 2023-12-16T07:30:48.280602+0000 mon.a (mon.0) 237 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-16T07:30:49.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:48 smithi007 bash[30436]: audit 2023-12-16T07:30:48.282757+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:49.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:48 smithi169 bash[25083]: cluster 2023-12-16T07:30:47.333955+0000 mgr.y (mgr.14152) 73 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:49.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:48 smithi169 bash[25083]: audit 2023-12-16T07:30:48.280602+0000 mon.a (mon.0) 237 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-16T07:30:49.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:48 smithi169 bash[25083]: audit 2023-12-16T07:30:48.282757+0000 mon.a (mon.0) 238 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:50.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:49 smithi007 bash[25009]: cephadm 2023-12-16T07:30:48.284428+0000 mgr.y (mgr.14152) 74 : cephadm [INF] Deploying daemon osd.0 on smithi007 2023-12-16T07:30:50.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:50 smithi007 bash[30436]: cephadm 2023-12-16T07:30:48.284428+0000 mgr.y (mgr.14152) 74 : cephadm [INF] Deploying daemon osd.0 on smithi007 2023-12-16T07:30:50.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:49 smithi169 bash[25083]: cephadm 2023-12-16T07:30:48.284428+0000 mgr.y (mgr.14152) 74 : cephadm [INF] Deploying daemon osd.0 on smithi007 2023-12-16T07:30:51.298 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:50 smithi007 bash[25009]: cluster 2023-12-16T07:30:49.334438+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:51.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:50 smithi007 bash[30436]: cluster 2023-12-16T07:30:49.334438+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:51.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:51 smithi169 bash[25083]: cluster 2023-12-16T07:30:49.334438+0000 mgr.y (mgr.14152) 75 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:53.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:53 smithi007 bash[25009]: cluster 2023-12-16T07:30:51.334989+0000 mgr.y (mgr.14152) 76 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:53.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:53 smithi007 bash[25009]: audit 2023-12-16T07:30:52.798304+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:53.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:53 smithi007 bash[25009]: audit 2023-12-16T07:30:52.803476+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:53.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:53 smithi007 bash[25009]: audit 2023-12-16T07:30:52.805161+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:53.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:53 smithi007 bash[25009]: audit 2023-12-16T07:30:52.806242+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:53.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:53 smithi007 bash[30436]: cluster 2023-12-16T07:30:51.334989+0000 mgr.y (mgr.14152) 76 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:53.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:53 smithi007 bash[30436]: audit 2023-12-16T07:30:52.798304+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:53.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:53 smithi007 bash[30436]: audit 2023-12-16T07:30:52.803476+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:53.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:53 smithi007 bash[30436]: audit 2023-12-16T07:30:52.805161+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:53.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:53 smithi007 bash[30436]: audit 2023-12-16T07:30:52.806242+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:53.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:53 smithi169 bash[25083]: cluster 2023-12-16T07:30:51.334989+0000 mgr.y (mgr.14152) 76 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:53.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:53 smithi169 bash[25083]: audit 2023-12-16T07:30:52.798304+0000 mon.a (mon.0) 239 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:53.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:53 smithi169 bash[25083]: audit 2023-12-16T07:30:52.803476+0000 mon.a (mon.0) 240 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:30:53.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:53 smithi169 bash[25083]: audit 2023-12-16T07:30:52.805161+0000 mon.a (mon.0) 241 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:30:53.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:53 smithi169 bash[25083]: audit 2023-12-16T07:30:52.806242+0000 mon.a (mon.0) 242 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:30:55.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:55 smithi007 bash[25009]: cluster 2023-12-16T07:30:53.335506+0000 mgr.y (mgr.14152) 77 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:55.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:55 smithi007 bash[30436]: cluster 2023-12-16T07:30:53.335506+0000 mgr.y (mgr.14152) 77 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:55.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:55 smithi169 bash[25083]: cluster 2023-12-16T07:30:53.335506+0000 mgr.y (mgr.14152) 77 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:57.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:57 smithi007 bash[25009]: cluster 2023-12-16T07:30:55.335981+0000 mgr.y (mgr.14152) 78 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:57.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:57 smithi007 bash[25009]: audit 2023-12-16T07:30:56.906422+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:57.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:57 smithi007 bash[25009]: audit 2023-12-16T07:30:56.913703+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:57.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:57 smithi007 bash[30436]: cluster 2023-12-16T07:30:55.335981+0000 mgr.y (mgr.14152) 78 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:57.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:57 smithi007 bash[30436]: audit 2023-12-16T07:30:56.906422+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:57.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:57 smithi007 bash[30436]: audit 2023-12-16T07:30:56.913703+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:57.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:57 smithi169 bash[25083]: cluster 2023-12-16T07:30:55.335981+0000 mgr.y (mgr.14152) 78 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:57.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:57 smithi169 bash[25083]: audit 2023-12-16T07:30:56.906422+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:57.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:57 smithi169 bash[25083]: audit 2023-12-16T07:30:56.913703+0000 mon.a (mon.0) 244 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:30:59.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:30:59 smithi007 bash[25009]: cluster 2023-12-16T07:30:57.336466+0000 mgr.y (mgr.14152) 79 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:59.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:30:59 smithi007 bash[30436]: cluster 2023-12-16T07:30:57.336466+0000 mgr.y (mgr.14152) 79 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:59.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:30:59 smithi169 bash[25083]: cluster 2023-12-16T07:30:57.336466+0000 mgr.y (mgr.14152) 79 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:30:59.709 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 0 on host 'smithi007' 2023-12-16T07:31:00.371 DEBUG:teuthology.orchestra.run.smithi007:osd.0> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.0.service 2023-12-16T07:31:00.373 INFO:tasks.cephadm:Deploying osd.1 on smithi007 with /dev/vg_nvme/lv_3... 2023-12-16T07:31:00.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-12-16T07:31:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:00 smithi007 bash[30436]: cluster 2023-12-16T07:30:59.336956+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:31:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:00 smithi007 bash[30436]: audit 2023-12-16T07:30:59.707423+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:00 smithi007 bash[30436]: audit 2023-12-16T07:30:59.737985+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:31:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:00 smithi007 bash[30436]: audit 2023-12-16T07:30:59.740236+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:00 smithi007 bash[30436]: audit 2023-12-16T07:30:59.741692+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:31:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:00 smithi007 bash[30436]: audit 2023-12-16T07:31:00.336299+0000 mon.a (mon.0) 249 : audit [INF] from='osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-16T07:31:00.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:00 smithi007 bash[25009]: cluster 2023-12-16T07:30:59.336956+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:31:00.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:00 smithi007 bash[25009]: audit 2023-12-16T07:30:59.707423+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:00.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:00 smithi007 bash[25009]: audit 2023-12-16T07:30:59.737985+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:31:00.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:00 smithi007 bash[25009]: audit 2023-12-16T07:30:59.740236+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:00.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:00 smithi007 bash[25009]: audit 2023-12-16T07:30:59.741692+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:31:00.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:00 smithi007 bash[25009]: audit 2023-12-16T07:31:00.336299+0000 mon.a (mon.0) 249 : audit [INF] from='osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-16T07:31:00.840 INFO:journalctl@ceph.osd.0.smithi007.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:31:01.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:00 smithi169 bash[25083]: cluster 2023-12-16T07:30:59.336956+0000 mgr.y (mgr.14152) 80 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:31:01.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:00 smithi169 bash[25083]: audit 2023-12-16T07:30:59.707423+0000 mon.a (mon.0) 245 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:01.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:00 smithi169 bash[25083]: audit 2023-12-16T07:30:59.737985+0000 mon.a (mon.0) 246 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:31:01.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:00 smithi169 bash[25083]: audit 2023-12-16T07:30:59.740236+0000 mon.a (mon.0) 247 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:01.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:00 smithi169 bash[25083]: audit 2023-12-16T07:30:59.741692+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:31:01.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:00 smithi169 bash[25083]: audit 2023-12-16T07:31:00.336299+0000 mon.a (mon.0) 249 : audit [INF] from='osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-16T07:31:02.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:02 smithi007 bash[25009]: audit 2023-12-16T07:31:01.022273+0000 mon.a (mon.0) 250 : audit [INF] from='osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-16T07:31:02.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:02 smithi007 bash[25009]: cluster 2023-12-16T07:31:01.022458+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-12-16T07:31:02.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:02 smithi007 bash[25009]: audit 2023-12-16T07:31:01.022734+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:31:02.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:02 smithi007 bash[25009]: audit 2023-12-16T07:31:01.023643+0000 mon.a (mon.0) 253 : audit [INF] from='osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:31:02.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:02 smithi007 bash[30436]: audit 2023-12-16T07:31:01.022273+0000 mon.a (mon.0) 250 : audit [INF] from='osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-16T07:31:02.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:02 smithi007 bash[30436]: cluster 2023-12-16T07:31:01.022458+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-12-16T07:31:02.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:02 smithi007 bash[30436]: audit 2023-12-16T07:31:01.022734+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:31:02.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:02 smithi007 bash[30436]: audit 2023-12-16T07:31:01.023643+0000 mon.a (mon.0) 253 : audit [INF] from='osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:31:02.317 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:31:02 smithi007 bash[33229]: debug 2023-12-16T07:31:02.027+0000 7faedccb2700 -1 osd.0 0 waiting for initial osdmap 2023-12-16T07:31:02.317 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:31:02 smithi007 bash[33229]: debug 2023-12-16T07:31:02.031+0000 7faed5e46700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:31:02.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:02 smithi169 bash[25083]: audit 2023-12-16T07:31:01.022273+0000 mon.a (mon.0) 250 : audit [INF] from='osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-16T07:31:02.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:02 smithi169 bash[25083]: cluster 2023-12-16T07:31:01.022458+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2023-12-16T07:31:02.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:02 smithi169 bash[25083]: audit 2023-12-16T07:31:01.022734+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:31:02.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:02 smithi169 bash[25083]: audit 2023-12-16T07:31:01.023643+0000 mon.a (mon.0) 253 : audit [INF] from='osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:31:03.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:03 smithi007 bash[30436]: cluster 2023-12-16T07:31:01.337385+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:31:03.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:03 smithi007 bash[30436]: audit 2023-12-16T07:31:02.023748+0000 mon.a (mon.0) 254 : audit [INF] from='osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-12-16T07:31:03.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:03 smithi007 bash[30436]: cluster 2023-12-16T07:31:02.023920+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2023-12-16T07:31:03.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:03 smithi007 bash[30436]: audit 2023-12-16T07:31:02.024703+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:31:03.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:03 smithi007 bash[30436]: audit 2023-12-16T07:31:02.034175+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:31:03.360 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:03 smithi007 bash[25009]: cluster 2023-12-16T07:31:01.337385+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:31:03.360 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:03 smithi007 bash[25009]: audit 2023-12-16T07:31:02.023748+0000 mon.a (mon.0) 254 : audit [INF] from='osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-12-16T07:31:03.360 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:03 smithi007 bash[25009]: cluster 2023-12-16T07:31:02.023920+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2023-12-16T07:31:03.360 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:03 smithi007 bash[25009]: audit 2023-12-16T07:31:02.024703+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:31:03.360 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:03 smithi007 bash[25009]: audit 2023-12-16T07:31:02.034175+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:31:03.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:03 smithi169 bash[25083]: cluster 2023-12-16T07:31:01.337385+0000 mgr.y (mgr.14152) 81 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:31:03.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:03 smithi169 bash[25083]: audit 2023-12-16T07:31:02.023748+0000 mon.a (mon.0) 254 : audit [INF] from='osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-12-16T07:31:03.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:03 smithi169 bash[25083]: cluster 2023-12-16T07:31:02.023920+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2023-12-16T07:31:03.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:03 smithi169 bash[25083]: audit 2023-12-16T07:31:02.024703+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:31:03.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:03 smithi169 bash[25083]: audit 2023-12-16T07:31:02.034175+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:31:04.306 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:04 smithi007 bash[30436]: cluster 2023-12-16T07:31:01.341202+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:31:04.307 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:04 smithi007 bash[30436]: cluster 2023-12-16T07:31:01.341458+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:31:04.307 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:04 smithi007 bash[30436]: cluster 2023-12-16T07:31:03.034115+0000 mon.a (mon.0) 258 : cluster [INF] osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011] boot 2023-12-16T07:31:04.307 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:04 smithi007 bash[30436]: cluster 2023-12-16T07:31:03.034260+0000 mon.a (mon.0) 259 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2023-12-16T07:31:04.307 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:04 smithi007 bash[30436]: audit 2023-12-16T07:31:03.035378+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:31:04.307 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:04 smithi007 bash[25009]: cluster 2023-12-16T07:31:01.341202+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:31:04.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:04 smithi007 bash[25009]: cluster 2023-12-16T07:31:01.341458+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:31:04.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:04 smithi007 bash[25009]: cluster 2023-12-16T07:31:03.034115+0000 mon.a (mon.0) 258 : cluster [INF] osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011] boot 2023-12-16T07:31:04.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:04 smithi007 bash[25009]: cluster 2023-12-16T07:31:03.034260+0000 mon.a (mon.0) 259 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2023-12-16T07:31:04.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:04 smithi007 bash[25009]: audit 2023-12-16T07:31:03.035378+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:31:04.407 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-12-16T07:31:04.408 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-12-16T07:31:04.408 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2023-12-16T07:31:04.408 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2023-12-16T07:31:04.409 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0178409 s, 588 MB/s 2023-12-16T07:31:04.409 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2023-12-16T07:31:04.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:04 smithi169 bash[25083]: cluster 2023-12-16T07:31:01.341202+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:31:04.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:04 smithi169 bash[25083]: cluster 2023-12-16T07:31:01.341458+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:31:04.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:04 smithi169 bash[25083]: cluster 2023-12-16T07:31:03.034115+0000 mon.a (mon.0) 258 : cluster [INF] osd.0 [v2:172.21.15.7:6802/3002763011,v1:172.21.15.7:6803/3002763011] boot 2023-12-16T07:31:04.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:04 smithi169 bash[25083]: cluster 2023-12-16T07:31:03.034260+0000 mon.a (mon.0) 259 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2023-12-16T07:31:04.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:04 smithi169 bash[25083]: audit 2023-12-16T07:31:03.035378+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:31:05.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:05 smithi007 bash[25009]: cluster 2023-12-16T07:31:03.337864+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:31:05.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:05 smithi007 bash[25009]: cluster 2023-12-16T07:31:04.047122+0000 mon.a (mon.0) 261 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2023-12-16T07:31:05.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:05 smithi007 bash[30436]: cluster 2023-12-16T07:31:03.337864+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:31:05.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:05 smithi007 bash[30436]: cluster 2023-12-16T07:31:04.047122+0000 mon.a (mon.0) 261 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2023-12-16T07:31:05.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:05 smithi169 bash[25083]: cluster 2023-12-16T07:31:03.337864+0000 mgr.y (mgr.14152) 82 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-16T07:31:05.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:05 smithi169 bash[25083]: cluster 2023-12-16T07:31:04.047122+0000 mon.a (mon.0) 261 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2023-12-16T07:31:06.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi007:vg_nvme/lv_3 2023-12-16T07:31:06.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:06 smithi007 bash[25009]: cluster 2023-12-16T07:31:05.338336+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:06.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:06 smithi007 bash[30436]: cluster 2023-12-16T07:31:05.338336+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:06.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:06 smithi169 bash[25083]: cluster 2023-12-16T07:31:05.338336+0000 mgr.y (mgr.14152) 83 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:09.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:08 smithi007 bash[25009]: cluster 2023-12-16T07:31:07.338866+0000 mgr.y (mgr.14152) 84 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:09.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:08 smithi007 bash[30436]: cluster 2023-12-16T07:31:07.338866+0000 mgr.y (mgr.14152) 84 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:09.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:08 smithi169 bash[25083]: cluster 2023-12-16T07:31:07.338866+0000 mgr.y (mgr.14152) 84 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:10.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:10 smithi169 bash[25083]: cephadm 2023-12-16T07:31:09.323883+0000 mgr.y (mgr.14152) 85 : cephadm [INF] Detected new or changed devices on smithi007 2023-12-16T07:31:10.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:10 smithi169 bash[25083]: audit 2023-12-16T07:31:09.332536+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:10.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:10 smithi169 bash[25083]: audit 2023-12-16T07:31:09.334167+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:31:10.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:10 smithi169 bash[25083]: cephadm 2023-12-16T07:31:09.335363+0000 mgr.y (mgr.14152) 86 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 16251M 2023-12-16T07:31:10.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:10 smithi169 bash[25083]: cluster 2023-12-16T07:31:09.339262+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:10.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:10 smithi169 bash[25083]: audit 2023-12-16T07:31:09.343214+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:10.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:10 smithi169 bash[25083]: audit 2023-12-16T07:31:09.350702+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:10.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:10 smithi169 bash[25083]: audit 2023-12-16T07:31:09.857171+0000 mgr.y (mgr.14152) 88 : audit [DBG] from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:31:10.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:10 smithi169 bash[25083]: audit 2023-12-16T07:31:09.863796+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:31:10.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:10 smithi169 bash[25083]: audit 2023-12-16T07:31:09.868303+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:31:10.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:10 smithi169 bash[25083]: audit 2023-12-16T07:31:09.869745+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:10.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[25009]: cephadm 2023-12-16T07:31:09.323883+0000 mgr.y (mgr.14152) 85 : cephadm [INF] Detected new or changed devices on smithi007 2023-12-16T07:31:10.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[25009]: audit 2023-12-16T07:31:09.332536+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:10.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[25009]: audit 2023-12-16T07:31:09.334167+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:31:10.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[25009]: cephadm 2023-12-16T07:31:09.335363+0000 mgr.y (mgr.14152) 86 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 16251M 2023-12-16T07:31:10.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[25009]: cluster 2023-12-16T07:31:09.339262+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:10.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[25009]: audit 2023-12-16T07:31:09.343214+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:10.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[25009]: audit 2023-12-16T07:31:09.350702+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:10.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[25009]: audit 2023-12-16T07:31:09.857171+0000 mgr.y (mgr.14152) 88 : audit [DBG] from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:31:10.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[25009]: audit 2023-12-16T07:31:09.863796+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:31:10.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[25009]: audit 2023-12-16T07:31:09.868303+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:31:10.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[25009]: audit 2023-12-16T07:31:09.869745+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:10.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[30436]: cephadm 2023-12-16T07:31:09.323883+0000 mgr.y (mgr.14152) 85 : cephadm [INF] Detected new or changed devices on smithi007 2023-12-16T07:31:10.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[30436]: audit 2023-12-16T07:31:09.332536+0000 mon.a (mon.0) 262 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:10.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[30436]: audit 2023-12-16T07:31:09.334167+0000 mon.a (mon.0) 263 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:31:10.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[30436]: cephadm 2023-12-16T07:31:09.335363+0000 mgr.y (mgr.14152) 86 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 16251M 2023-12-16T07:31:10.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[30436]: cluster 2023-12-16T07:31:09.339262+0000 mgr.y (mgr.14152) 87 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:10.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[30436]: audit 2023-12-16T07:31:09.343214+0000 mon.a (mon.0) 264 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:10.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[30436]: audit 2023-12-16T07:31:09.350702+0000 mon.a (mon.0) 265 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:10.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[30436]: audit 2023-12-16T07:31:09.857171+0000 mgr.y (mgr.14152) 88 : audit [DBG] from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:31:10.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[30436]: audit 2023-12-16T07:31:09.863796+0000 mon.a (mon.0) 266 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:31:10.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[30436]: audit 2023-12-16T07:31:09.868303+0000 mon.a (mon.0) 267 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:31:10.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:10 smithi007 bash[30436]: audit 2023-12-16T07:31:09.869745+0000 mon.a (mon.0) 268 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:13.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:12 smithi007 bash[30436]: cluster 2023-12-16T07:31:11.339734+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:13.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:12 smithi007 bash[25009]: cluster 2023-12-16T07:31:11.339734+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:13.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:12 smithi169 bash[25083]: cluster 2023-12-16T07:31:11.339734+0000 mgr.y (mgr.14152) 89 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:15.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:14 smithi007 bash[25009]: cluster 2023-12-16T07:31:13.340238+0000 mgr.y (mgr.14152) 90 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:15.254 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:14 smithi007 bash[30436]: cluster 2023-12-16T07:31:13.340238+0000 mgr.y (mgr.14152) 90 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:15.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:14 smithi169 bash[25083]: cluster 2023-12-16T07:31:13.340238+0000 mgr.y (mgr.14152) 90 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:16.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:16 smithi007 bash[25009]: audit 2023-12-16T07:31:15.387565+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:31:16.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:16 smithi007 bash[25009]: audit 2023-12-16T07:31:15.389392+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:31:16.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:16 smithi007 bash[30436]: audit 2023-12-16T07:31:15.387565+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:31:16.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:16 smithi007 bash[30436]: audit 2023-12-16T07:31:15.389392+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:31:16.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:16 smithi169 bash[25083]: audit 2023-12-16T07:31:15.387565+0000 mon.a (mon.0) 269 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:31:16.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:16 smithi169 bash[25083]: audit 2023-12-16T07:31:15.389392+0000 mon.a (mon.0) 270 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:31:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:17 smithi007 bash[25009]: cluster 2023-12-16T07:31:15.340613+0000 mgr.y (mgr.14152) 91 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:17 smithi007 bash[25009]: audit 2023-12-16T07:31:16.133286+0000 mon.a (mon.0) 271 : audit [INF] from='client.? 172.21.15.7:0/3015472467' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b52565e1-b236-4f11-8356-c6e90f476298"}]: dispatch 2023-12-16T07:31:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:17 smithi007 bash[25009]: audit 2023-12-16T07:31:16.138461+0000 mon.a (mon.0) 272 : audit [INF] from='client.? 172.21.15.7:0/3015472467' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b52565e1-b236-4f11-8356-c6e90f476298"}]': finished 2023-12-16T07:31:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:17 smithi007 bash[25009]: cluster 2023-12-16T07:31:16.138537+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2023-12-16T07:31:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:17 smithi007 bash[25009]: audit 2023-12-16T07:31:16.138785+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:17 smithi007 bash[25009]: audit 2023-12-16T07:31:16.860892+0000 mon.c (mon.1) 3 : audit [DBG] from='client.? 172.21.15.7:0/3246725996' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:31:17.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:17 smithi007 bash[30436]: cluster 2023-12-16T07:31:15.340613+0000 mgr.y (mgr.14152) 91 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:17.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:17 smithi007 bash[30436]: audit 2023-12-16T07:31:16.133286+0000 mon.a (mon.0) 271 : audit [INF] from='client.? 172.21.15.7:0/3015472467' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b52565e1-b236-4f11-8356-c6e90f476298"}]: dispatch 2023-12-16T07:31:17.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:17 smithi007 bash[30436]: audit 2023-12-16T07:31:16.138461+0000 mon.a (mon.0) 272 : audit [INF] from='client.? 172.21.15.7:0/3015472467' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b52565e1-b236-4f11-8356-c6e90f476298"}]': finished 2023-12-16T07:31:17.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:17 smithi007 bash[30436]: cluster 2023-12-16T07:31:16.138537+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2023-12-16T07:31:17.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:17 smithi007 bash[30436]: audit 2023-12-16T07:31:16.138785+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:17.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:17 smithi007 bash[30436]: audit 2023-12-16T07:31:16.860892+0000 mon.c (mon.1) 3 : audit [DBG] from='client.? 172.21.15.7:0/3246725996' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:31:17.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:17 smithi169 bash[25083]: cluster 2023-12-16T07:31:15.340613+0000 mgr.y (mgr.14152) 91 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:17.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:17 smithi169 bash[25083]: audit 2023-12-16T07:31:16.133286+0000 mon.a (mon.0) 271 : audit [INF] from='client.? 172.21.15.7:0/3015472467' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b52565e1-b236-4f11-8356-c6e90f476298"}]: dispatch 2023-12-16T07:31:17.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:17 smithi169 bash[25083]: audit 2023-12-16T07:31:16.138461+0000 mon.a (mon.0) 272 : audit [INF] from='client.? 172.21.15.7:0/3015472467' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b52565e1-b236-4f11-8356-c6e90f476298"}]': finished 2023-12-16T07:31:17.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:17 smithi169 bash[25083]: cluster 2023-12-16T07:31:16.138537+0000 mon.a (mon.0) 273 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2023-12-16T07:31:17.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:17 smithi169 bash[25083]: audit 2023-12-16T07:31:16.138785+0000 mon.a (mon.0) 274 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:17.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:17 smithi169 bash[25083]: audit 2023-12-16T07:31:16.860892+0000 mon.c (mon.1) 3 : audit [DBG] from='client.? 172.21.15.7:0/3246725996' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:31:19.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:19 smithi007 bash[30436]: cluster 2023-12-16T07:31:17.341064+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:19.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:19 smithi007 bash[25009]: cluster 2023-12-16T07:31:17.341064+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:19.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:19 smithi169 bash[25083]: cluster 2023-12-16T07:31:17.341064+0000 mgr.y (mgr.14152) 92 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:21.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:21 smithi007 bash[25009]: cluster 2023-12-16T07:31:19.341516+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:21.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:21 smithi007 bash[30436]: cluster 2023-12-16T07:31:19.341516+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:21.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:21 smithi169 bash[25083]: cluster 2023-12-16T07:31:19.341516+0000 mgr.y (mgr.14152) 93 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:23.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:23 smithi007 bash[25009]: cluster 2023-12-16T07:31:21.341986+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:23.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:23 smithi007 bash[30436]: cluster 2023-12-16T07:31:21.341986+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:23.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:23 smithi169 bash[25083]: cluster 2023-12-16T07:31:21.341986+0000 mgr.y (mgr.14152) 94 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:25.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:25 smithi007 bash[25009]: cluster 2023-12-16T07:31:23.342512+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:25.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:25 smithi007 bash[30436]: cluster 2023-12-16T07:31:23.342512+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:25.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:25 smithi169 bash[25083]: cluster 2023-12-16T07:31:23.342512+0000 mgr.y (mgr.14152) 95 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:27.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:27 smithi007 bash[25009]: cluster 2023-12-16T07:31:25.343078+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:27.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:27 smithi007 bash[25009]: audit 2023-12-16T07:31:26.621706+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-16T07:31:27.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:27 smithi007 bash[25009]: audit 2023-12-16T07:31:26.623250+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:27.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:27 smithi007 bash[30436]: cluster 2023-12-16T07:31:25.343078+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:27.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:27 smithi007 bash[30436]: audit 2023-12-16T07:31:26.621706+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-16T07:31:27.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:27 smithi007 bash[30436]: audit 2023-12-16T07:31:26.623250+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:27.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:27 smithi169 bash[25083]: cluster 2023-12-16T07:31:25.343078+0000 mgr.y (mgr.14152) 96 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:27 smithi169 bash[25083]: audit 2023-12-16T07:31:26.621706+0000 mon.a (mon.0) 275 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-16T07:31:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:27 smithi169 bash[25083]: audit 2023-12-16T07:31:26.623250+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:28.168 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:28 smithi007 bash[30436]: cephadm 2023-12-16T07:31:26.624304+0000 mgr.y (mgr.14152) 97 : cephadm [INF] Deploying daemon osd.1 on smithi007 2023-12-16T07:31:28.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:28 smithi007 bash[25009]: cephadm 2023-12-16T07:31:26.624304+0000 mgr.y (mgr.14152) 97 : cephadm [INF] Deploying daemon osd.1 on smithi007 2023-12-16T07:31:28.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:28 smithi169 bash[25083]: cephadm 2023-12-16T07:31:26.624304+0000 mgr.y (mgr.14152) 97 : cephadm [INF] Deploying daemon osd.1 on smithi007 2023-12-16T07:31:29.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:29 smithi007 bash[30436]: cluster 2023-12-16T07:31:27.343746+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:29.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:29 smithi007 bash[25009]: cluster 2023-12-16T07:31:27.343746+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:29.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:29 smithi169 bash[25083]: cluster 2023-12-16T07:31:27.343746+0000 mgr.y (mgr.14152) 98 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:31.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:31 smithi007 bash[25009]: cluster 2023-12-16T07:31:29.344297+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:31.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:31 smithi007 bash[25009]: audit 2023-12-16T07:31:30.968538+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:31.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:31 smithi007 bash[25009]: audit 2023-12-16T07:31:30.971697+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:31:31.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:31 smithi007 bash[25009]: audit 2023-12-16T07:31:30.974538+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:31.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:31 smithi007 bash[25009]: audit 2023-12-16T07:31:30.976151+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:31:31.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:31 smithi007 bash[30436]: cluster 2023-12-16T07:31:29.344297+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:31.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:31 smithi007 bash[30436]: audit 2023-12-16T07:31:30.968538+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:31.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:31 smithi007 bash[30436]: audit 2023-12-16T07:31:30.971697+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:31:31.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:31 smithi007 bash[30436]: audit 2023-12-16T07:31:30.974538+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:31.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:31 smithi007 bash[30436]: audit 2023-12-16T07:31:30.976151+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:31:31.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:31 smithi169 bash[25083]: cluster 2023-12-16T07:31:29.344297+0000 mgr.y (mgr.14152) 99 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:31.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:31 smithi169 bash[25083]: audit 2023-12-16T07:31:30.968538+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:31.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:31 smithi169 bash[25083]: audit 2023-12-16T07:31:30.971697+0000 mon.a (mon.0) 278 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:31:31.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:31 smithi169 bash[25083]: audit 2023-12-16T07:31:30.974538+0000 mon.a (mon.0) 279 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:31.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:31 smithi169 bash[25083]: audit 2023-12-16T07:31:30.976151+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:31:33.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:33 smithi007 bash[30436]: cluster 2023-12-16T07:31:31.344778+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:33.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:33 smithi007 bash[25009]: cluster 2023-12-16T07:31:31.344778+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:33.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:33 smithi169 bash[25083]: cluster 2023-12-16T07:31:31.344778+0000 mgr.y (mgr.14152) 100 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:35.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:35 smithi007 bash[25009]: cluster 2023-12-16T07:31:33.345319+0000 mgr.y (mgr.14152) 101 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:35.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:35 smithi007 bash[25009]: audit 2023-12-16T07:31:34.954515+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:35.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:35 smithi007 bash[30436]: cluster 2023-12-16T07:31:33.345319+0000 mgr.y (mgr.14152) 101 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:35.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:35 smithi007 bash[30436]: audit 2023-12-16T07:31:34.954515+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:35.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:35 smithi169 bash[25083]: cluster 2023-12-16T07:31:33.345319+0000 mgr.y (mgr.14152) 101 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:35.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:35 smithi169 bash[25083]: audit 2023-12-16T07:31:34.954515+0000 mon.a (mon.0) 281 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:36.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:36 smithi007 bash[25009]: audit 2023-12-16T07:31:35.231697+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:36.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:36 smithi007 bash[25009]: cluster 2023-12-16T07:31:35.345808+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:36.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:36 smithi007 bash[30436]: audit 2023-12-16T07:31:35.231697+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:36.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:36 smithi007 bash[30436]: cluster 2023-12-16T07:31:35.345808+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:36.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:36 smithi169 bash[25083]: audit 2023-12-16T07:31:35.231697+0000 mon.a (mon.0) 282 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:36.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:36 smithi169 bash[25083]: cluster 2023-12-16T07:31:35.345808+0000 mgr.y (mgr.14152) 102 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:37.689 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 1 on host 'smithi007' 2023-12-16T07:31:38.268 DEBUG:teuthology.orchestra.run.smithi007:osd.1> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.1.service 2023-12-16T07:31:38.271 INFO:tasks.cephadm:Deploying osd.2 on smithi007 with /dev/vg_nvme/lv_2... 2023-12-16T07:31:38.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-12-16T07:31:38.401 INFO:journalctl@ceph.osd.1.smithi007.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:31:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:38 smithi007 bash[25009]: cluster 2023-12-16T07:31:37.346308+0000 mgr.y (mgr.14152) 103 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:38 smithi007 bash[25009]: audit 2023-12-16T07:31:37.687563+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:38 smithi007 bash[25009]: audit 2023-12-16T07:31:37.704798+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:31:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:38 smithi007 bash[25009]: audit 2023-12-16T07:31:37.707002+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:38 smithi007 bash[25009]: audit 2023-12-16T07:31:37.708502+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:31:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:38 smithi007 bash[25009]: audit 2023-12-16T07:31:38.161781+0000 mon.a (mon.0) 287 : audit [INF] from='osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-16T07:31:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:38 smithi007 bash[30436]: cluster 2023-12-16T07:31:37.346308+0000 mgr.y (mgr.14152) 103 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:38 smithi007 bash[30436]: audit 2023-12-16T07:31:37.687563+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:38 smithi007 bash[30436]: audit 2023-12-16T07:31:37.704798+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:31:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:38 smithi007 bash[30436]: audit 2023-12-16T07:31:37.707002+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:38 smithi007 bash[30436]: audit 2023-12-16T07:31:37.708502+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:31:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:38 smithi007 bash[30436]: audit 2023-12-16T07:31:38.161781+0000 mon.a (mon.0) 287 : audit [INF] from='osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-16T07:31:39.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:38 smithi169 bash[25083]: cluster 2023-12-16T07:31:37.346308+0000 mgr.y (mgr.14152) 103 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:39.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:38 smithi169 bash[25083]: audit 2023-12-16T07:31:37.687563+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:39.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:38 smithi169 bash[25083]: audit 2023-12-16T07:31:37.704798+0000 mon.a (mon.0) 284 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:31:39.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:38 smithi169 bash[25083]: audit 2023-12-16T07:31:37.707002+0000 mon.a (mon.0) 285 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:39.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:38 smithi169 bash[25083]: audit 2023-12-16T07:31:37.708502+0000 mon.a (mon.0) 286 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:31:39.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:38 smithi169 bash[25083]: audit 2023-12-16T07:31:38.161781+0000 mon.a (mon.0) 287 : audit [INF] from='osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-16T07:31:40.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:40 smithi007 bash[25009]: audit 2023-12-16T07:31:39.008785+0000 mon.a (mon.0) 288 : audit [INF] from='osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-16T07:31:40.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:40 smithi007 bash[25009]: cluster 2023-12-16T07:31:39.008901+0000 mon.a (mon.0) 289 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2023-12-16T07:31:40.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:40 smithi007 bash[25009]: audit 2023-12-16T07:31:39.009221+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:40.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:40 smithi007 bash[25009]: audit 2023-12-16T07:31:39.010213+0000 mon.a (mon.0) 291 : audit [INF] from='osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:31:40.316 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:31:40 smithi007 bash[36352]: debug 2023-12-16T07:31:40.015+0000 7f52d9147700 -1 osd.1 0 waiting for initial osdmap 2023-12-16T07:31:40.317 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:31:40 smithi007 bash[36352]: debug 2023-12-16T07:31:40.019+0000 7f52d22db700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:31:40.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:40 smithi007 bash[30436]: audit 2023-12-16T07:31:39.008785+0000 mon.a (mon.0) 288 : audit [INF] from='osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-16T07:31:40.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:40 smithi007 bash[30436]: cluster 2023-12-16T07:31:39.008901+0000 mon.a (mon.0) 289 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2023-12-16T07:31:40.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:40 smithi007 bash[30436]: audit 2023-12-16T07:31:39.009221+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:40.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:40 smithi007 bash[30436]: audit 2023-12-16T07:31:39.010213+0000 mon.a (mon.0) 291 : audit [INF] from='osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:31:40.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:40 smithi169 bash[25083]: audit 2023-12-16T07:31:39.008785+0000 mon.a (mon.0) 288 : audit [INF] from='osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-16T07:31:40.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:40 smithi169 bash[25083]: cluster 2023-12-16T07:31:39.008901+0000 mon.a (mon.0) 289 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2023-12-16T07:31:40.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:40 smithi169 bash[25083]: audit 2023-12-16T07:31:39.009221+0000 mon.a (mon.0) 290 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:40.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:40 smithi169 bash[25083]: audit 2023-12-16T07:31:39.010213+0000 mon.a (mon.0) 291 : audit [INF] from='osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:31:41.028 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:41 smithi007 bash[30436]: cluster 2023-12-16T07:31:39.346841+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:41.029 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:41 smithi007 bash[30436]: audit 2023-12-16T07:31:40.011376+0000 mon.a (mon.0) 292 : audit [INF] from='osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-12-16T07:31:41.029 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:41 smithi007 bash[30436]: cluster 2023-12-16T07:31:40.011483+0000 mon.a (mon.0) 293 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2023-12-16T07:31:41.029 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:41 smithi007 bash[30436]: audit 2023-12-16T07:31:40.012177+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:41.029 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:41 smithi007 bash[30436]: audit 2023-12-16T07:31:40.019729+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:41.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:41 smithi007 bash[25009]: cluster 2023-12-16T07:31:39.346841+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:41.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:41 smithi007 bash[25009]: audit 2023-12-16T07:31:40.011376+0000 mon.a (mon.0) 292 : audit [INF] from='osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-12-16T07:31:41.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:41 smithi007 bash[25009]: cluster 2023-12-16T07:31:40.011483+0000 mon.a (mon.0) 293 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2023-12-16T07:31:41.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:41 smithi007 bash[25009]: audit 2023-12-16T07:31:40.012177+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:41.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:41 smithi007 bash[25009]: audit 2023-12-16T07:31:40.019729+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:41.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:41 smithi169 bash[25083]: cluster 2023-12-16T07:31:39.346841+0000 mgr.y (mgr.14152) 104 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:41.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:41 smithi169 bash[25083]: audit 2023-12-16T07:31:40.011376+0000 mon.a (mon.0) 292 : audit [INF] from='osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-12-16T07:31:41.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:41 smithi169 bash[25083]: cluster 2023-12-16T07:31:40.011483+0000 mon.a (mon.0) 293 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2023-12-16T07:31:41.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:41 smithi169 bash[25083]: audit 2023-12-16T07:31:40.012177+0000 mon.a (mon.0) 294 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:41.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:41 smithi169 bash[25083]: audit 2023-12-16T07:31:40.019729+0000 mon.a (mon.0) 295 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:42.188 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:42 smithi007 bash[30436]: cluster 2023-12-16T07:31:39.201709+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:31:42.188 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:42 smithi007 bash[30436]: cluster 2023-12-16T07:31:39.201914+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:31:42.188 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:42 smithi007 bash[30436]: cluster 2023-12-16T07:31:41.023275+0000 mon.a (mon.0) 296 : cluster [INF] osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404] boot 2023-12-16T07:31:42.188 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:42 smithi007 bash[30436]: cluster 2023-12-16T07:31:41.023370+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2023-12-16T07:31:42.188 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:42 smithi007 bash[30436]: audit 2023-12-16T07:31:41.024409+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:42.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:42 smithi007 bash[25009]: cluster 2023-12-16T07:31:39.201709+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:31:42.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:42 smithi007 bash[25009]: cluster 2023-12-16T07:31:39.201914+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:31:42.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:42 smithi007 bash[25009]: cluster 2023-12-16T07:31:41.023275+0000 mon.a (mon.0) 296 : cluster [INF] osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404] boot 2023-12-16T07:31:42.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:42 smithi007 bash[25009]: cluster 2023-12-16T07:31:41.023370+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2023-12-16T07:31:42.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:42 smithi007 bash[25009]: audit 2023-12-16T07:31:41.024409+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:42.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:42 smithi169 bash[25083]: cluster 2023-12-16T07:31:39.201709+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:31:42.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:42 smithi169 bash[25083]: cluster 2023-12-16T07:31:39.201914+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:31:42.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:42 smithi169 bash[25083]: cluster 2023-12-16T07:31:41.023275+0000 mon.a (mon.0) 296 : cluster [INF] osd.1 [v2:172.21.15.7:6810/4053186404,v1:172.21.15.7:6811/4053186404] boot 2023-12-16T07:31:42.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:42 smithi169 bash[25083]: cluster 2023-12-16T07:31:41.023370+0000 mon.a (mon.0) 297 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2023-12-16T07:31:42.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:42 smithi169 bash[25083]: audit 2023-12-16T07:31:41.024409+0000 mon.a (mon.0) 298 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:31:42.503 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-12-16T07:31:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-12-16T07:31:42.504 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2023-12-16T07:31:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2023-12-16T07:31:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113002 s, 928 MB/s 2023-12-16T07:31:42.504 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 2023-12-16T07:31:42.504 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2023-12-16T07:31:43.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:43 smithi007 bash[30436]: cluster 2023-12-16T07:31:41.347326+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:43.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:43 smithi007 bash[30436]: cluster 2023-12-16T07:31:42.038496+0000 mon.a (mon.0) 299 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2023-12-16T07:31:43.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:43 smithi007 bash[25009]: cluster 2023-12-16T07:31:41.347326+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:43.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:43 smithi007 bash[25009]: cluster 2023-12-16T07:31:42.038496+0000 mon.a (mon.0) 299 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2023-12-16T07:31:43.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:43 smithi169 bash[25083]: cluster 2023-12-16T07:31:41.347326+0000 mgr.y (mgr.14152) 105 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2023-12-16T07:31:43.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:43 smithi169 bash[25083]: cluster 2023-12-16T07:31:42.038496+0000 mon.a (mon.0) 299 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2023-12-16T07:31:43.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi007:vg_nvme/lv_2 2023-12-16T07:31:45.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:45 smithi007 bash[25009]: cluster 2023-12-16T07:31:43.347917+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:45.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:45 smithi007 bash[30436]: cluster 2023-12-16T07:31:43.347917+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:45.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:45 smithi169 bash[25083]: cluster 2023-12-16T07:31:43.347917+0000 mgr.y (mgr.14152) 106 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:47.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:47 smithi007 bash[25009]: cluster 2023-12-16T07:31:45.348486+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:47.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:47 smithi007 bash[30436]: cluster 2023-12-16T07:31:45.348486+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:47.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:47 smithi169 bash[25083]: cluster 2023-12-16T07:31:45.348486+0000 mgr.y (mgr.14152) 107 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:48.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:48 smithi169 bash[25083]: cephadm 2023-12-16T07:31:47.336551+0000 mgr.y (mgr.14152) 108 : cephadm [INF] Detected new or changed devices on smithi007 2023-12-16T07:31:48.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:48 smithi169 bash[25083]: audit 2023-12-16T07:31:47.345228+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:48.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:48 smithi169 bash[25083]: audit 2023-12-16T07:31:47.347437+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:31:48.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:48 smithi169 bash[25083]: cluster 2023-12-16T07:31:47.348900+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:48.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:48 smithi169 bash[25083]: audit 2023-12-16T07:31:47.349153+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:31:48.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:48 smithi169 bash[25083]: cephadm 2023-12-16T07:31:47.350582+0000 mgr.y (mgr.14152) 110 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 8125M 2023-12-16T07:31:48.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:48 smithi169 bash[25083]: audit 2023-12-16T07:31:47.357713+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:48.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:48 smithi169 bash[25083]: audit 2023-12-16T07:31:47.364188+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:48.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:48 smithi169 bash[25083]: audit 2023-12-16T07:31:47.829855+0000 mgr.y (mgr.14152) 111 : audit [DBG] from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:31:48.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:48 smithi169 bash[25083]: audit 2023-12-16T07:31:47.836031+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:31:48.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:48 smithi169 bash[25083]: audit 2023-12-16T07:31:47.840045+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:31:48.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:48 smithi169 bash[25083]: audit 2023-12-16T07:31:47.841550+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:48.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[25009]: cephadm 2023-12-16T07:31:47.336551+0000 mgr.y (mgr.14152) 108 : cephadm [INF] Detected new or changed devices on smithi007 2023-12-16T07:31:48.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[25009]: audit 2023-12-16T07:31:47.345228+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:48.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[25009]: audit 2023-12-16T07:31:47.347437+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:31:48.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[25009]: cluster 2023-12-16T07:31:47.348900+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:48.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[25009]: audit 2023-12-16T07:31:47.349153+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:31:48.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[25009]: cephadm 2023-12-16T07:31:47.350582+0000 mgr.y (mgr.14152) 110 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 8125M 2023-12-16T07:31:48.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[25009]: audit 2023-12-16T07:31:47.357713+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:48.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[25009]: audit 2023-12-16T07:31:47.364188+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:48.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[25009]: audit 2023-12-16T07:31:47.829855+0000 mgr.y (mgr.14152) 111 : audit [DBG] from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:31:48.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[25009]: audit 2023-12-16T07:31:47.836031+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:31:48.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[25009]: audit 2023-12-16T07:31:47.840045+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:31:48.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[25009]: audit 2023-12-16T07:31:47.841550+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:48.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[30436]: cephadm 2023-12-16T07:31:47.336551+0000 mgr.y (mgr.14152) 108 : cephadm [INF] Detected new or changed devices on smithi007 2023-12-16T07:31:48.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[30436]: audit 2023-12-16T07:31:47.345228+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:48.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[30436]: audit 2023-12-16T07:31:47.347437+0000 mon.a (mon.0) 301 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:31:48.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[30436]: cluster 2023-12-16T07:31:47.348900+0000 mgr.y (mgr.14152) 109 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:48.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[30436]: audit 2023-12-16T07:31:47.349153+0000 mon.a (mon.0) 302 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:31:48.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[30436]: cephadm 2023-12-16T07:31:47.350582+0000 mgr.y (mgr.14152) 110 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 8125M 2023-12-16T07:31:48.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[30436]: audit 2023-12-16T07:31:47.357713+0000 mon.a (mon.0) 303 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:48.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[30436]: audit 2023-12-16T07:31:47.364188+0000 mon.a (mon.0) 304 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:31:48.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[30436]: audit 2023-12-16T07:31:47.829855+0000 mgr.y (mgr.14152) 111 : audit [DBG] from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:31:48.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[30436]: audit 2023-12-16T07:31:47.836031+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:31:48.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[30436]: audit 2023-12-16T07:31:47.840045+0000 mon.a (mon.0) 306 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:31:48.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:48 smithi007 bash[30436]: audit 2023-12-16T07:31:47.841550+0000 mon.a (mon.0) 307 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:31:50.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:50 smithi169 bash[25083]: cluster 2023-12-16T07:31:49.349379+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:50.815 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:50 smithi007 bash[30436]: cluster 2023-12-16T07:31:49.349379+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:50.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:50 smithi007 bash[25009]: cluster 2023-12-16T07:31:49.349379+0000 mgr.y (mgr.14152) 112 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:53.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:53 smithi007 bash[25009]: cluster 2023-12-16T07:31:51.349874+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:53.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:53 smithi007 bash[30436]: cluster 2023-12-16T07:31:51.349874+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:53.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:53 smithi169 bash[25083]: cluster 2023-12-16T07:31:51.349874+0000 mgr.y (mgr.14152) 113 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:54.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:54 smithi007 bash[25009]: cluster 2023-12-16T07:31:53.350238+0000 mgr.y (mgr.14152) 114 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:54.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:54 smithi007 bash[25009]: audit 2023-12-16T07:31:54.005967+0000 mon.c (mon.1) 4 : audit [INF] from='client.? 172.21.15.7:0/3131970692' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a43c2e0-1837-4880-b904-5ea6041d45a2"}]: dispatch 2023-12-16T07:31:54.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:54 smithi007 bash[25009]: audit 2023-12-16T07:31:54.006757+0000 mon.a (mon.0) 308 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a43c2e0-1837-4880-b904-5ea6041d45a2"}]: dispatch 2023-12-16T07:31:54.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:54 smithi007 bash[25009]: audit 2023-12-16T07:31:54.016863+0000 mon.a (mon.0) 309 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a43c2e0-1837-4880-b904-5ea6041d45a2"}]': finished 2023-12-16T07:31:54.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:54 smithi007 bash[25009]: cluster 2023-12-16T07:31:54.016926+0000 mon.a (mon.0) 310 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2023-12-16T07:31:54.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:54 smithi007 bash[25009]: audit 2023-12-16T07:31:54.017061+0000 mon.a (mon.0) 311 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:31:54.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:54 smithi007 bash[30436]: cluster 2023-12-16T07:31:53.350238+0000 mgr.y (mgr.14152) 114 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:54.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:54 smithi007 bash[30436]: audit 2023-12-16T07:31:54.005967+0000 mon.c (mon.1) 4 : audit [INF] from='client.? 172.21.15.7:0/3131970692' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a43c2e0-1837-4880-b904-5ea6041d45a2"}]: dispatch 2023-12-16T07:31:54.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:54 smithi007 bash[30436]: audit 2023-12-16T07:31:54.006757+0000 mon.a (mon.0) 308 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a43c2e0-1837-4880-b904-5ea6041d45a2"}]: dispatch 2023-12-16T07:31:54.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:54 smithi007 bash[30436]: audit 2023-12-16T07:31:54.016863+0000 mon.a (mon.0) 309 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a43c2e0-1837-4880-b904-5ea6041d45a2"}]': finished 2023-12-16T07:31:54.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:54 smithi007 bash[30436]: cluster 2023-12-16T07:31:54.016926+0000 mon.a (mon.0) 310 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2023-12-16T07:31:54.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:54 smithi007 bash[30436]: audit 2023-12-16T07:31:54.017061+0000 mon.a (mon.0) 311 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:31:54.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:54 smithi169 bash[25083]: cluster 2023-12-16T07:31:53.350238+0000 mgr.y (mgr.14152) 114 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:54.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:54 smithi169 bash[25083]: audit 2023-12-16T07:31:54.005967+0000 mon.c (mon.1) 4 : audit [INF] from='client.? 172.21.15.7:0/3131970692' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a43c2e0-1837-4880-b904-5ea6041d45a2"}]: dispatch 2023-12-16T07:31:54.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:54 smithi169 bash[25083]: audit 2023-12-16T07:31:54.006757+0000 mon.a (mon.0) 308 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a43c2e0-1837-4880-b904-5ea6041d45a2"}]: dispatch 2023-12-16T07:31:54.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:54 smithi169 bash[25083]: audit 2023-12-16T07:31:54.016863+0000 mon.a (mon.0) 309 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a43c2e0-1837-4880-b904-5ea6041d45a2"}]': finished 2023-12-16T07:31:54.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:54 smithi169 bash[25083]: cluster 2023-12-16T07:31:54.016926+0000 mon.a (mon.0) 310 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2023-12-16T07:31:54.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:54 smithi169 bash[25083]: audit 2023-12-16T07:31:54.017061+0000 mon.a (mon.0) 311 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:31:55.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:55 smithi007 bash[25009]: audit 2023-12-16T07:31:54.763128+0000 mon.a (mon.0) 312 : audit [DBG] from='client.? 172.21.15.7:0/2864661998' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:31:55.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:55 smithi007 bash[30436]: audit 2023-12-16T07:31:54.763128+0000 mon.a (mon.0) 312 : audit [DBG] from='client.? 172.21.15.7:0/2864661998' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:31:55.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:55 smithi169 bash[25083]: audit 2023-12-16T07:31:54.763128+0000 mon.a (mon.0) 312 : audit [DBG] from='client.? 172.21.15.7:0/2864661998' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:31:56.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:56 smithi169 bash[25083]: cluster 2023-12-16T07:31:55.350760+0000 mgr.y (mgr.14152) 115 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:56.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:56 smithi007 bash[25009]: cluster 2023-12-16T07:31:55.350760+0000 mgr.y (mgr.14152) 115 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:56.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:56 smithi007 bash[30436]: cluster 2023-12-16T07:31:55.350760+0000 mgr.y (mgr.14152) 115 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:59.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:31:59 smithi007 bash[25009]: cluster 2023-12-16T07:31:57.351326+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:59.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:31:59 smithi007 bash[30436]: cluster 2023-12-16T07:31:57.351326+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:31:59.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:31:59 smithi169 bash[25083]: cluster 2023-12-16T07:31:57.351326+0000 mgr.y (mgr.14152) 116 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:01.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:01 smithi007 bash[25009]: cluster 2023-12-16T07:31:59.351845+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:01.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:01 smithi007 bash[30436]: cluster 2023-12-16T07:31:59.351845+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:01.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:01 smithi169 bash[25083]: cluster 2023-12-16T07:31:59.351845+0000 mgr.y (mgr.14152) 117 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:03.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:03 smithi007 bash[30436]: cluster 2023-12-16T07:32:01.352432+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:03.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:03 smithi007 bash[25009]: cluster 2023-12-16T07:32:01.352432+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:03.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:03 smithi169 bash[25083]: cluster 2023-12-16T07:32:01.352432+0000 mgr.y (mgr.14152) 118 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:05.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:05 smithi007 bash[25009]: cluster 2023-12-16T07:32:03.353035+0000 mgr.y (mgr.14152) 119 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:05.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:05 smithi007 bash[25009]: audit 2023-12-16T07:32:04.685642+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-16T07:32:05.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:05 smithi007 bash[25009]: audit 2023-12-16T07:32:04.687307+0000 mon.a (mon.0) 314 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:05.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:05 smithi007 bash[30436]: cluster 2023-12-16T07:32:03.353035+0000 mgr.y (mgr.14152) 119 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:05.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:05 smithi007 bash[30436]: audit 2023-12-16T07:32:04.685642+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-16T07:32:05.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:05 smithi007 bash[30436]: audit 2023-12-16T07:32:04.687307+0000 mon.a (mon.0) 314 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:05.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:05 smithi169 bash[25083]: cluster 2023-12-16T07:32:03.353035+0000 mgr.y (mgr.14152) 119 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:05.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:05 smithi169 bash[25083]: audit 2023-12-16T07:32:04.685642+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-16T07:32:05.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:05 smithi169 bash[25083]: audit 2023-12-16T07:32:04.687307+0000 mon.a (mon.0) 314 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:06.212 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:06 smithi007 bash[25009]: cephadm 2023-12-16T07:32:04.688637+0000 mgr.y (mgr.14152) 120 : cephadm [INF] Deploying daemon osd.2 on smithi007 2023-12-16T07:32:06.213 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:06 smithi007 bash[30436]: cephadm 2023-12-16T07:32:04.688637+0000 mgr.y (mgr.14152) 120 : cephadm [INF] Deploying daemon osd.2 on smithi007 2023-12-16T07:32:06.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:06 smithi169 bash[25083]: cephadm 2023-12-16T07:32:04.688637+0000 mgr.y (mgr.14152) 120 : cephadm [INF] Deploying daemon osd.2 on smithi007 2023-12-16T07:32:07.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:07 smithi007 bash[30436]: cluster 2023-12-16T07:32:05.353684+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:07.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:07 smithi007 bash[25009]: cluster 2023-12-16T07:32:05.353684+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:07.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:07 smithi169 bash[25083]: cluster 2023-12-16T07:32:05.353684+0000 mgr.y (mgr.14152) 121 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:09.291 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:09 smithi169 bash[25083]: cluster 2023-12-16T07:32:07.354229+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:09.291 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:09 smithi169 bash[25083]: audit 2023-12-16T07:32:08.977371+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:09.291 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:09 smithi169 bash[25083]: audit 2023-12-16T07:32:08.989364+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:32:09.292 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:09 smithi169 bash[25083]: audit 2023-12-16T07:32:08.991578+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:09.292 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:09 smithi169 bash[25083]: audit 2023-12-16T07:32:08.993048+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:32:09.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:09 smithi007 bash[30436]: cluster 2023-12-16T07:32:07.354229+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:09.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:09 smithi007 bash[30436]: audit 2023-12-16T07:32:08.977371+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:09.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:09 smithi007 bash[30436]: audit 2023-12-16T07:32:08.989364+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:32:09.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:09 smithi007 bash[30436]: audit 2023-12-16T07:32:08.991578+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:09.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:09 smithi007 bash[30436]: audit 2023-12-16T07:32:08.993048+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:32:09.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:09 smithi007 bash[25009]: cluster 2023-12-16T07:32:07.354229+0000 mgr.y (mgr.14152) 122 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:09.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:09 smithi007 bash[25009]: audit 2023-12-16T07:32:08.977371+0000 mon.a (mon.0) 315 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:09.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:09 smithi007 bash[25009]: audit 2023-12-16T07:32:08.989364+0000 mon.a (mon.0) 316 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:32:09.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:09 smithi007 bash[25009]: audit 2023-12-16T07:32:08.991578+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:09.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:09 smithi007 bash[25009]: audit 2023-12-16T07:32:08.993048+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:32:11.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:11 smithi007 bash[25009]: cluster 2023-12-16T07:32:09.354877+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:11.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:11 smithi007 bash[30436]: cluster 2023-12-16T07:32:09.354877+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:11.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:11 smithi169 bash[25083]: cluster 2023-12-16T07:32:09.354877+0000 mgr.y (mgr.14152) 123 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:13.311 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:13 smithi007 bash[25009]: cluster 2023-12-16T07:32:11.355468+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:13.311 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:13 smithi007 bash[25009]: audit 2023-12-16T07:32:13.037333+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:13.311 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:13 smithi007 bash[30436]: cluster 2023-12-16T07:32:11.355468+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:13.312 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:13 smithi007 bash[30436]: audit 2023-12-16T07:32:13.037333+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:13.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:13 smithi169 bash[25083]: cluster 2023-12-16T07:32:11.355468+0000 mgr.y (mgr.14152) 124 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:13.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:13 smithi169 bash[25083]: audit 2023-12-16T07:32:13.037333+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:14.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:14 smithi007 bash[25009]: audit 2023-12-16T07:32:13.045488+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:14.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:14 smithi007 bash[30436]: audit 2023-12-16T07:32:13.045488+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:14.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:14 smithi169 bash[25083]: audit 2023-12-16T07:32:13.045488+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:15.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:15 smithi007 bash[30436]: cluster 2023-12-16T07:32:13.356074+0000 mgr.y (mgr.14152) 125 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:15.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:15 smithi007 bash[25009]: cluster 2023-12-16T07:32:13.356074+0000 mgr.y (mgr.14152) 125 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:15.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:15 smithi169 bash[25083]: cluster 2023-12-16T07:32:13.356074+0000 mgr.y (mgr.14152) 125 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:16.056 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 2 on host 'smithi007' 2023-12-16T07:32:16.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:16 smithi007 bash[30436]: audit 2023-12-16T07:32:15.389753+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:32:16.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:16 smithi007 bash[30436]: audit 2023-12-16T07:32:15.392048+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:32:16.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:16 smithi007 bash[30436]: audit 2023-12-16T07:32:16.054433+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:16.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:16 smithi007 bash[25009]: audit 2023-12-16T07:32:15.389753+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:32:16.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:16 smithi007 bash[25009]: audit 2023-12-16T07:32:15.392048+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:32:16.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:16 smithi007 bash[25009]: audit 2023-12-16T07:32:16.054433+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:16.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:16 smithi169 bash[25083]: audit 2023-12-16T07:32:15.389753+0000 mon.a (mon.0) 321 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:32:16.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:16 smithi169 bash[25083]: audit 2023-12-16T07:32:15.392048+0000 mon.a (mon.0) 322 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:32:16.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:16 smithi169 bash[25083]: audit 2023-12-16T07:32:16.054433+0000 mon.a (mon.0) 323 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:16.820 DEBUG:teuthology.orchestra.run.smithi007:osd.2> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.2.service 2023-12-16T07:32:16.823 INFO:tasks.cephadm:Deploying osd.3 on smithi007 with /dev/vg_nvme/lv_1... 2023-12-16T07:32:16.823 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-12-16T07:32:16.946 INFO:journalctl@ceph.osd.2.smithi007.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:32:17.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:17 smithi007 bash[25009]: cluster 2023-12-16T07:32:15.356626+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:17 smithi007 bash[25009]: audit 2023-12-16T07:32:16.077884+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:32:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:17 smithi007 bash[25009]: audit 2023-12-16T07:32:16.079963+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:17 smithi007 bash[25009]: audit 2023-12-16T07:32:16.081280+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:32:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:17 smithi007 bash[25009]: audit 2023-12-16T07:32:16.445553+0000 mon.a (mon.0) 327 : audit [INF] from='osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-16T07:32:17.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:17 smithi007 bash[30436]: cluster 2023-12-16T07:32:15.356626+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:17.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:17 smithi007 bash[30436]: audit 2023-12-16T07:32:16.077884+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:32:17.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:17 smithi007 bash[30436]: audit 2023-12-16T07:32:16.079963+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:17.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:17 smithi007 bash[30436]: audit 2023-12-16T07:32:16.081280+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:32:17.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:17 smithi007 bash[30436]: audit 2023-12-16T07:32:16.445553+0000 mon.a (mon.0) 327 : audit [INF] from='osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-16T07:32:17.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:17 smithi169 bash[25083]: cluster 2023-12-16T07:32:15.356626+0000 mgr.y (mgr.14152) 126 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:17.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:17 smithi169 bash[25083]: audit 2023-12-16T07:32:16.077884+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:32:17.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:17 smithi169 bash[25083]: audit 2023-12-16T07:32:16.079963+0000 mon.a (mon.0) 325 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:17.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:17 smithi169 bash[25083]: audit 2023-12-16T07:32:16.081280+0000 mon.a (mon.0) 326 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:32:17.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:17 smithi169 bash[25083]: audit 2023-12-16T07:32:16.445553+0000 mon.a (mon.0) 327 : audit [INF] from='osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-16T07:32:18.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[30436]: audit 2023-12-16T07:32:17.071842+0000 mon.a (mon.0) 328 : audit [INF] from='osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-16T07:32:18.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[30436]: cluster 2023-12-16T07:32:17.071968+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2023-12-16T07:32:18.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[30436]: audit 2023-12-16T07:32:17.072694+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:18.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[30436]: audit 2023-12-16T07:32:17.073684+0000 mon.a (mon.0) 331 : audit [INF] from='osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:32:18.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[30436]: cluster 2023-12-16T07:32:17.357143+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:18.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[30436]: audit 2023-12-16T07:32:18.073326+0000 mon.a (mon.0) 332 : audit [INF] from='osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-12-16T07:32:18.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[30436]: cluster 2023-12-16T07:32:18.073470+0000 mon.a (mon.0) 333 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2023-12-16T07:32:18.294 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[39502]: debug 2023-12-16T07:32:18.075+0000 7f492b4ed700 -1 osd.2 0 waiting for initial osdmap 2023-12-16T07:32:18.295 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[39502]: debug 2023-12-16T07:32:18.083+0000 7f4923e80700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:32:18.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[25009]: audit 2023-12-16T07:32:17.071842+0000 mon.a (mon.0) 328 : audit [INF] from='osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-16T07:32:18.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[25009]: cluster 2023-12-16T07:32:17.071968+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2023-12-16T07:32:18.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[25009]: audit 2023-12-16T07:32:17.072694+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:18.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[25009]: audit 2023-12-16T07:32:17.073684+0000 mon.a (mon.0) 331 : audit [INF] from='osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:32:18.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[25009]: cluster 2023-12-16T07:32:17.357143+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:18.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[25009]: audit 2023-12-16T07:32:18.073326+0000 mon.a (mon.0) 332 : audit [INF] from='osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-12-16T07:32:18.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:18 smithi007 bash[25009]: cluster 2023-12-16T07:32:18.073470+0000 mon.a (mon.0) 333 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2023-12-16T07:32:18.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:18 smithi169 bash[25083]: audit 2023-12-16T07:32:17.071842+0000 mon.a (mon.0) 328 : audit [INF] from='osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-16T07:32:18.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:18 smithi169 bash[25083]: cluster 2023-12-16T07:32:17.071968+0000 mon.a (mon.0) 329 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2023-12-16T07:32:18.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:18 smithi169 bash[25083]: audit 2023-12-16T07:32:17.072694+0000 mon.a (mon.0) 330 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:18.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:18 smithi169 bash[25083]: audit 2023-12-16T07:32:17.073684+0000 mon.a (mon.0) 331 : audit [INF] from='osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:32:18.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:18 smithi169 bash[25083]: cluster 2023-12-16T07:32:17.357143+0000 mgr.y (mgr.14152) 127 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:18.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:18 smithi169 bash[25083]: audit 2023-12-16T07:32:18.073326+0000 mon.a (mon.0) 332 : audit [INF] from='osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-12-16T07:32:18.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:18 smithi169 bash[25083]: cluster 2023-12-16T07:32:18.073470+0000 mon.a (mon.0) 333 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2023-12-16T07:32:19.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:19 smithi007 bash[25009]: audit 2023-12-16T07:32:18.073718+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:19.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:19 smithi007 bash[25009]: audit 2023-12-16T07:32:18.086748+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:19.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:19 smithi007 bash[30436]: audit 2023-12-16T07:32:18.073718+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:19.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:19 smithi007 bash[30436]: audit 2023-12-16T07:32:18.086748+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:19.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:19 smithi169 bash[25083]: audit 2023-12-16T07:32:18.073718+0000 mon.a (mon.0) 334 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:19.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:19 smithi169 bash[25083]: audit 2023-12-16T07:32:18.086748+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:20.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:20 smithi169 bash[25083]: cluster 2023-12-16T07:32:17.444459+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:32:20.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:20 smithi169 bash[25083]: cluster 2023-12-16T07:32:17.444666+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:32:20.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:20 smithi169 bash[25083]: audit 2023-12-16T07:32:19.086063+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:20.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:20 smithi169 bash[25083]: cluster 2023-12-16T07:32:19.092261+0000 mon.a (mon.0) 337 : cluster [INF] osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438] boot 2023-12-16T07:32:20.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:20 smithi169 bash[25083]: cluster 2023-12-16T07:32:19.092320+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2023-12-16T07:32:20.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:20 smithi169 bash[25083]: audit 2023-12-16T07:32:19.093225+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:20.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:20 smithi169 bash[25083]: cluster 2023-12-16T07:32:19.357670+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:20.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:20 smithi169 bash[25083]: audit 2023-12-16T07:32:19.468143+0000 mon.a (mon.0) 340 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-12-16T07:32:20.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[30436]: cluster 2023-12-16T07:32:17.444459+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:32:20.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[30436]: cluster 2023-12-16T07:32:17.444666+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:32:20.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[30436]: audit 2023-12-16T07:32:19.086063+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:20.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[30436]: cluster 2023-12-16T07:32:19.092261+0000 mon.a (mon.0) 337 : cluster [INF] osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438] boot 2023-12-16T07:32:20.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[30436]: cluster 2023-12-16T07:32:19.092320+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2023-12-16T07:32:20.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[30436]: audit 2023-12-16T07:32:19.093225+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:20.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[30436]: cluster 2023-12-16T07:32:19.357670+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:20.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[30436]: audit 2023-12-16T07:32:19.468143+0000 mon.a (mon.0) 340 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-12-16T07:32:20.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[25009]: cluster 2023-12-16T07:32:17.444459+0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:32:20.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[25009]: cluster 2023-12-16T07:32:17.444666+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:32:20.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[25009]: audit 2023-12-16T07:32:19.086063+0000 mon.a (mon.0) 336 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:20.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[25009]: cluster 2023-12-16T07:32:19.092261+0000 mon.a (mon.0) 337 : cluster [INF] osd.2 [v2:172.21.15.7:6818/2173677438,v1:172.21.15.7:6819/2173677438] boot 2023-12-16T07:32:20.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[25009]: cluster 2023-12-16T07:32:19.092320+0000 mon.a (mon.0) 338 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2023-12-16T07:32:20.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[25009]: audit 2023-12-16T07:32:19.093225+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:32:20.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[25009]: cluster 2023-12-16T07:32:19.357670+0000 mgr.y (mgr.14152) 128 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-12-16T07:32:20.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:20 smithi007 bash[25009]: audit 2023-12-16T07:32:19.468143+0000 mon.a (mon.0) 340 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2023-12-16T07:32:20.789 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-12-16T07:32:20.789 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-12-16T07:32:20.789 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2023-12-16T07:32:20.789 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2023-12-16T07:32:20.789 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0202615 s, 518 MB/s 2023-12-16T07:32:20.789 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2023-12-16T07:32:21.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:21 smithi169 bash[25083]: audit 2023-12-16T07:32:20.097259+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-12-16T07:32:21.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:21 smithi169 bash[25083]: cluster 2023-12-16T07:32:20.097366+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2023-12-16T07:32:21.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:21 smithi169 bash[25083]: audit 2023-12-16T07:32:20.098819+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-12-16T07:32:21.474 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:21 smithi007 bash[30436]: audit 2023-12-16T07:32:20.097259+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-12-16T07:32:21.474 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:21 smithi007 bash[30436]: cluster 2023-12-16T07:32:20.097366+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2023-12-16T07:32:21.474 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:21 smithi007 bash[30436]: audit 2023-12-16T07:32:20.098819+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-12-16T07:32:21.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:21 smithi007 bash[25009]: audit 2023-12-16T07:32:20.097259+0000 mon.a (mon.0) 341 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2023-12-16T07:32:21.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:21 smithi007 bash[25009]: cluster 2023-12-16T07:32:20.097366+0000 mon.a (mon.0) 342 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2023-12-16T07:32:21.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:21 smithi007 bash[25009]: audit 2023-12-16T07:32:20.098819+0000 mon.a (mon.0) 343 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-12-16T07:32:22.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:22 smithi169 bash[25083]: audit 2023-12-16T07:32:21.108981+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-12-16T07:32:22.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:22 smithi169 bash[25083]: cluster 2023-12-16T07:32:21.109132+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2023-12-16T07:32:22.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:22 smithi169 bash[25083]: cluster 2023-12-16T07:32:21.358238+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v103: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:22.464 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:22 smithi007 bash[30436]: audit 2023-12-16T07:32:21.108981+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-12-16T07:32:22.464 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:22 smithi007 bash[30436]: cluster 2023-12-16T07:32:21.109132+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2023-12-16T07:32:22.464 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:22 smithi007 bash[30436]: cluster 2023-12-16T07:32:21.358238+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v103: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:22.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:22 smithi007 bash[25009]: audit 2023-12-16T07:32:21.108981+0000 mon.a (mon.0) 344 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-12-16T07:32:22.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:22 smithi007 bash[25009]: cluster 2023-12-16T07:32:21.109132+0000 mon.a (mon.0) 345 : cluster [DBG] osdmap e20: 3 total, 3 up, 3 in 2023-12-16T07:32:22.465 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:22 smithi007 bash[25009]: cluster 2023-12-16T07:32:21.358238+0000 mgr.y (mgr.14152) 129 : cluster [DBG] pgmap v103: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:22.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi007:vg_nvme/lv_1 2023-12-16T07:32:23.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:23 smithi169 bash[25083]: cluster 2023-12-16T07:32:22.124034+0000 mon.a (mon.0) 346 : cluster [DBG] mgrmap e15: y(active, since 3m), standbys: x 2023-12-16T07:32:23.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:23 smithi169 bash[25083]: cluster 2023-12-16T07:32:22.124090+0000 mon.a (mon.0) 347 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2023-12-16T07:32:23.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:23 smithi169 bash[25083]: audit 2023-12-16T07:32:22.853567+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:23.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:23 smithi007 bash[30436]: cluster 2023-12-16T07:32:22.124034+0000 mon.a (mon.0) 346 : cluster [DBG] mgrmap e15: y(active, since 3m), standbys: x 2023-12-16T07:32:23.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:23 smithi007 bash[30436]: cluster 2023-12-16T07:32:22.124090+0000 mon.a (mon.0) 347 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2023-12-16T07:32:23.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:23 smithi007 bash[30436]: audit 2023-12-16T07:32:22.853567+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:23.442 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:23 smithi007 bash[25009]: cluster 2023-12-16T07:32:22.124034+0000 mon.a (mon.0) 346 : cluster [DBG] mgrmap e15: y(active, since 3m), standbys: x 2023-12-16T07:32:23.442 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:23 smithi007 bash[25009]: cluster 2023-12-16T07:32:22.124090+0000 mon.a (mon.0) 347 : cluster [DBG] osdmap e21: 3 total, 3 up, 3 in 2023-12-16T07:32:23.442 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:23 smithi007 bash[25009]: audit 2023-12-16T07:32:22.853567+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:24.201 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:24 smithi007 bash[30436]: cluster 2023-12-16T07:32:23.358797+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:24.201 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:24 smithi007 bash[25009]: cluster 2023-12-16T07:32:23.358797+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:24.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:24 smithi169 bash[25083]: cluster 2023-12-16T07:32:23.358797+0000 mgr.y (mgr.14152) 130 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:26.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[25009]: audit 2023-12-16T07:32:24.685937+0000 mon.a (mon.0) 349 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T07:32:26.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[25009]: audit 2023-12-16T07:32:25.581519+0000 mon.a (mon.0) 350 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T07:32:26.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[25009]: audit 2023-12-16T07:32:25.581730+0000 mon.a (mon.0) 351 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:32:26.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[25009]: audit 2023-12-16T07:32:25.582955+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:32:26.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[25009]: audit 2023-12-16T07:32:25.583146+0000 mon.a (mon.0) 353 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:32:26.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[25009]: audit 2023-12-16T07:32:25.586096+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:32:26.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[25009]: audit 2023-12-16T07:32:25.586447+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:32:26.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[25009]: audit 2023-12-16T07:32:25.586734+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:32:26.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[30436]: audit 2023-12-16T07:32:24.685937+0000 mon.a (mon.0) 349 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T07:32:26.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[30436]: audit 2023-12-16T07:32:25.581519+0000 mon.a (mon.0) 350 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T07:32:26.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[30436]: audit 2023-12-16T07:32:25.581730+0000 mon.a (mon.0) 351 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:32:26.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[30436]: audit 2023-12-16T07:32:25.582955+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:32:26.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[30436]: audit 2023-12-16T07:32:25.583146+0000 mon.a (mon.0) 353 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:32:26.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[30436]: audit 2023-12-16T07:32:25.586096+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:32:26.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[30436]: audit 2023-12-16T07:32:25.586447+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:32:26.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:26 smithi007 bash[30436]: audit 2023-12-16T07:32:25.586734+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:32:26.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:26 smithi169 bash[25083]: audit 2023-12-16T07:32:24.685937+0000 mon.a (mon.0) 349 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T07:32:26.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:26 smithi169 bash[25083]: audit 2023-12-16T07:32:25.581519+0000 mon.a (mon.0) 350 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T07:32:26.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:26 smithi169 bash[25083]: audit 2023-12-16T07:32:25.581730+0000 mon.a (mon.0) 351 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:32:26.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:26 smithi169 bash[25083]: audit 2023-12-16T07:32:25.582955+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:32:26.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:26 smithi169 bash[25083]: audit 2023-12-16T07:32:25.583146+0000 mon.a (mon.0) 353 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:32:26.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:26 smithi169 bash[25083]: audit 2023-12-16T07:32:25.586096+0000 mon.a (mon.0) 354 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:32:26.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:26 smithi169 bash[25083]: audit 2023-12-16T07:32:25.586447+0000 mon.a (mon.0) 355 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:32:26.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:26 smithi169 bash[25083]: audit 2023-12-16T07:32:25.586734+0000 mon.a (mon.0) 356 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:32:27.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: cluster 2023-12-16T07:32:25.359354+0000 mgr.y (mgr.14152) 131 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:27.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:25.585732+0000 mon.c (mon.1) 5 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T07:32:27.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: cephadm 2023-12-16T07:32:25.825621+0000 mgr.y (mgr.14152) 132 : cephadm [INF] Detected new or changed devices on smithi007 2023-12-16T07:32:27.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:26.425001+0000 mgr.y (mgr.14152) 133 : audit [DBG] from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:32:27.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:26.431528+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:32:27.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:26.435757+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:32:27.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:26.436722+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:27.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:26.485570+0000 mon.c (mon.1) 6 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T07:32:27.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:26.495196+0000 mon.b (mon.2) 2 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T07:32:27.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:26.497593+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:27.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:26.497949+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:32:27.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:26.498487+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:32:27.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:26.498867+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:32:27.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:26.499261+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:32:27.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:26.500221+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:32:27.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:26.501096+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:32:27.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: cephadm 2023-12-16T07:32:26.501774+0000 mgr.y (mgr.14152) 134 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 5417M 2023-12-16T07:32:27.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:27.376396+0000 mon.b (mon.2) 3 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T07:32:27.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:27.387773+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:27.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:27.399904+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:27.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[25009]: audit 2023-12-16T07:32:27.411854+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: cluster 2023-12-16T07:32:25.359354+0000 mgr.y (mgr.14152) 131 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:25.585732+0000 mon.c (mon.1) 5 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T07:32:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: cephadm 2023-12-16T07:32:25.825621+0000 mgr.y (mgr.14152) 132 : cephadm [INF] Detected new or changed devices on smithi007 2023-12-16T07:32:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:26.425001+0000 mgr.y (mgr.14152) 133 : audit [DBG] from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:32:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:26.431528+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:32:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:26.435757+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:32:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:26.436722+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:26.485570+0000 mon.c (mon.1) 6 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T07:32:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:26.495196+0000 mon.b (mon.2) 2 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T07:32:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:26.497593+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:26.497949+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:32:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:26.498487+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:32:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:26.498867+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:32:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:26.499261+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:32:27.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:26.500221+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:32:27.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:26.501096+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:32:27.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: cephadm 2023-12-16T07:32:26.501774+0000 mgr.y (mgr.14152) 134 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 5417M 2023-12-16T07:32:27.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:27.376396+0000 mon.b (mon.2) 3 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T07:32:27.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:27.387773+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:27.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:27.399904+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:27.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:27 smithi007 bash[30436]: audit 2023-12-16T07:32:27.411854+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:27.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: cluster 2023-12-16T07:32:25.359354+0000 mgr.y (mgr.14152) 131 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:27.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:25.585732+0000 mon.c (mon.1) 5 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T07:32:27.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: cephadm 2023-12-16T07:32:25.825621+0000 mgr.y (mgr.14152) 132 : cephadm [INF] Detected new or changed devices on smithi007 2023-12-16T07:32:27.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:26.425001+0000 mgr.y (mgr.14152) 133 : audit [DBG] from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:32:27.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:26.431528+0000 mon.a (mon.0) 357 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:32:27.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:26.435757+0000 mon.a (mon.0) 358 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:32:27.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:26.436722+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:27.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:26.485570+0000 mon.c (mon.1) 6 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T07:32:27.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:26.495196+0000 mon.b (mon.2) 2 : audit [INF] from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-12-16T07:32:27.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:26.497593+0000 mon.a (mon.0) 360 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:27.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:26.497949+0000 mon.a (mon.0) 361 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:32:27.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:26.498487+0000 mon.a (mon.0) 362 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:32:27.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:26.498867+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:32:27.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:26.499261+0000 mon.a (mon.0) 364 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:32:27.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:26.500221+0000 mon.a (mon.0) 365 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:32:27.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:26.501096+0000 mon.a (mon.0) 366 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:32:27.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: cephadm 2023-12-16T07:32:26.501774+0000 mgr.y (mgr.14152) 134 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 5417M 2023-12-16T07:32:27.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:27.376396+0000 mon.b (mon.2) 3 : audit [INF] from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-12-16T07:32:27.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:27.387773+0000 mon.a (mon.0) 367 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:27.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:27.399904+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:27.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:27 smithi169 bash[25083]: audit 2023-12-16T07:32:27.411854+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:28.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:28 smithi007 bash[25009]: cluster 2023-12-16T07:32:27.359964+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-12-16T07:32:28.815 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:28 smithi007 bash[30436]: cluster 2023-12-16T07:32:27.359964+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-12-16T07:32:28.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:28 smithi169 bash[25083]: cluster 2023-12-16T07:32:27.359964+0000 mgr.y (mgr.14152) 135 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-12-16T07:32:30.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:30 smithi007 bash[25009]: cluster 2023-12-16T07:32:29.360552+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 72 KiB/s wr, 4 op/s 2023-12-16T07:32:30.815 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:30 smithi007 bash[30436]: cluster 2023-12-16T07:32:29.360552+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 72 KiB/s wr, 4 op/s 2023-12-16T07:32:30.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:30 smithi169 bash[25083]: cluster 2023-12-16T07:32:29.360552+0000 mgr.y (mgr.14152) 136 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 72 KiB/s wr, 4 op/s 2023-12-16T07:32:33.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:33 smithi007 bash[30436]: cluster 2023-12-16T07:32:31.361160+0000 mgr.y (mgr.14152) 137 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 996 B/s rd, 56 KiB/s wr, 3 op/s 2023-12-16T07:32:33.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:33 smithi007 bash[30436]: audit 2023-12-16T07:32:32.629673+0000 mon.a (mon.0) 370 : audit [INF] from='client.? 172.21.15.7:0/1580193651' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4776e261-fa8a-4652-a32f-83710dda5dbf"}]: dispatch 2023-12-16T07:32:33.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:33 smithi007 bash[30436]: audit 2023-12-16T07:32:32.637316+0000 mon.a (mon.0) 371 : audit [INF] from='client.? 172.21.15.7:0/1580193651' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4776e261-fa8a-4652-a32f-83710dda5dbf"}]': finished 2023-12-16T07:32:33.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:33 smithi007 bash[30436]: cluster 2023-12-16T07:32:32.637464+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2023-12-16T07:32:33.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:33 smithi007 bash[30436]: audit 2023-12-16T07:32:32.637779+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:33.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:33 smithi007 bash[25009]: cluster 2023-12-16T07:32:31.361160+0000 mgr.y (mgr.14152) 137 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 996 B/s rd, 56 KiB/s wr, 3 op/s 2023-12-16T07:32:33.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:33 smithi007 bash[25009]: audit 2023-12-16T07:32:32.629673+0000 mon.a (mon.0) 370 : audit [INF] from='client.? 172.21.15.7:0/1580193651' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4776e261-fa8a-4652-a32f-83710dda5dbf"}]: dispatch 2023-12-16T07:32:33.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:33 smithi007 bash[25009]: audit 2023-12-16T07:32:32.637316+0000 mon.a (mon.0) 371 : audit [INF] from='client.? 172.21.15.7:0/1580193651' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4776e261-fa8a-4652-a32f-83710dda5dbf"}]': finished 2023-12-16T07:32:33.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:33 smithi007 bash[25009]: cluster 2023-12-16T07:32:32.637464+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2023-12-16T07:32:33.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:33 smithi007 bash[25009]: audit 2023-12-16T07:32:32.637779+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:33.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:33 smithi169 bash[25083]: cluster 2023-12-16T07:32:31.361160+0000 mgr.y (mgr.14152) 137 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 996 B/s rd, 56 KiB/s wr, 3 op/s 2023-12-16T07:32:33.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:33 smithi169 bash[25083]: audit 2023-12-16T07:32:32.629673+0000 mon.a (mon.0) 370 : audit [INF] from='client.? 172.21.15.7:0/1580193651' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4776e261-fa8a-4652-a32f-83710dda5dbf"}]: dispatch 2023-12-16T07:32:33.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:33 smithi169 bash[25083]: audit 2023-12-16T07:32:32.637316+0000 mon.a (mon.0) 371 : audit [INF] from='client.? 172.21.15.7:0/1580193651' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4776e261-fa8a-4652-a32f-83710dda5dbf"}]': finished 2023-12-16T07:32:33.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:33 smithi169 bash[25083]: cluster 2023-12-16T07:32:32.637464+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2023-12-16T07:32:33.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:33 smithi169 bash[25083]: audit 2023-12-16T07:32:32.637779+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:34.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:34 smithi007 bash[25009]: audit 2023-12-16T07:32:33.360897+0000 mon.a (mon.0) 374 : audit [DBG] from='client.? 172.21.15.7:0/3695506018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:32:34.515 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:34 smithi007 bash[30436]: audit 2023-12-16T07:32:33.360897+0000 mon.a (mon.0) 374 : audit [DBG] from='client.? 172.21.15.7:0/3695506018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:32:34.515 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:34 smithi169 bash[25083]: audit 2023-12-16T07:32:33.360897+0000 mon.a (mon.0) 374 : audit [DBG] from='client.? 172.21.15.7:0/3695506018' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:32:35.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:35 smithi007 bash[25009]: cluster 2023-12-16T07:32:33.361787+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-12-16T07:32:35.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:35 smithi007 bash[30436]: cluster 2023-12-16T07:32:33.361787+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-12-16T07:32:35.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:35 smithi169 bash[25083]: cluster 2023-12-16T07:32:33.361787+0000 mgr.y (mgr.14152) 138 : cluster [DBG] pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-12-16T07:32:37.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:37 smithi007 bash[25009]: cluster 2023-12-16T07:32:35.362393+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-12-16T07:32:37.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:37 smithi007 bash[30436]: cluster 2023-12-16T07:32:35.362393+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-12-16T07:32:37.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:37 smithi169 bash[25083]: cluster 2023-12-16T07:32:35.362393+0000 mgr.y (mgr.14152) 139 : cluster [DBG] pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-12-16T07:32:39.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:39 smithi007 bash[30436]: cluster 2023-12-16T07:32:37.363038+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-12-16T07:32:39.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:39 smithi007 bash[25009]: cluster 2023-12-16T07:32:37.363038+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-12-16T07:32:39.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:39 smithi169 bash[25083]: cluster 2023-12-16T07:32:37.363038+0000 mgr.y (mgr.14152) 140 : cluster [DBG] pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-12-16T07:32:41.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:41 smithi007 bash[30436]: cluster 2023-12-16T07:32:39.363736+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-12-16T07:32:41.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:41 smithi007 bash[25009]: cluster 2023-12-16T07:32:39.363736+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-12-16T07:32:41.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:41 smithi169 bash[25083]: cluster 2023-12-16T07:32:39.363736+0000 mgr.y (mgr.14152) 141 : cluster [DBG] pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-12-16T07:32:43.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:43 smithi007 bash[25009]: cluster 2023-12-16T07:32:41.364385+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-12-16T07:32:43.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:43 smithi007 bash[30436]: cluster 2023-12-16T07:32:41.364385+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-12-16T07:32:43.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:43 smithi169 bash[25083]: cluster 2023-12-16T07:32:41.364385+0000 mgr.y (mgr.14152) 142 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2023-12-16T07:32:44.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:44 smithi007 bash[25009]: audit 2023-12-16T07:32:43.413260+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-16T07:32:44.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:44 smithi007 bash[25009]: audit 2023-12-16T07:32:43.414993+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:44.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:44 smithi007 bash[30436]: audit 2023-12-16T07:32:43.413260+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-16T07:32:44.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:44 smithi007 bash[30436]: audit 2023-12-16T07:32:43.414993+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:44.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:44 smithi169 bash[25083]: audit 2023-12-16T07:32:43.413260+0000 mon.a (mon.0) 375 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-16T07:32:44.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:44 smithi169 bash[25083]: audit 2023-12-16T07:32:43.414993+0000 mon.a (mon.0) 376 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:45.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:45 smithi007 bash[25009]: cluster 2023-12-16T07:32:43.364994+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:45.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:45 smithi007 bash[25009]: cephadm 2023-12-16T07:32:43.416426+0000 mgr.y (mgr.14152) 144 : cephadm [INF] Deploying daemon osd.3 on smithi007 2023-12-16T07:32:45.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:45 smithi007 bash[30436]: cluster 2023-12-16T07:32:43.364994+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:45.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:45 smithi007 bash[30436]: cephadm 2023-12-16T07:32:43.416426+0000 mgr.y (mgr.14152) 144 : cephadm [INF] Deploying daemon osd.3 on smithi007 2023-12-16T07:32:45.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:45 smithi169 bash[25083]: cluster 2023-12-16T07:32:43.364994+0000 mgr.y (mgr.14152) 143 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:45.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:45 smithi169 bash[25083]: cephadm 2023-12-16T07:32:43.416426+0000 mgr.y (mgr.14152) 144 : cephadm [INF] Deploying daemon osd.3 on smithi007 2023-12-16T07:32:47.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:47 smithi007 bash[30436]: cluster 2023-12-16T07:32:45.365618+0000 mgr.y (mgr.14152) 145 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:47.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:47 smithi007 bash[25009]: cluster 2023-12-16T07:32:45.365618+0000 mgr.y (mgr.14152) 145 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:47.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:47 smithi169 bash[25083]: cluster 2023-12-16T07:32:45.365618+0000 mgr.y (mgr.14152) 145 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:48.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:48 smithi169 bash[25083]: cluster 2023-12-16T07:32:47.366169+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:48.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:48 smithi169 bash[25083]: audit 2023-12-16T07:32:47.679398+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:48.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:48 smithi169 bash[25083]: audit 2023-12-16T07:32:47.719100+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:32:48.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:48 smithi169 bash[25083]: audit 2023-12-16T07:32:47.720805+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:48.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:48 smithi169 bash[25083]: audit 2023-12-16T07:32:47.722069+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:32:49.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:48 smithi007 bash[30436]: cluster 2023-12-16T07:32:47.366169+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:49.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:48 smithi007 bash[30436]: audit 2023-12-16T07:32:47.679398+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:49.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:48 smithi007 bash[30436]: audit 2023-12-16T07:32:47.719100+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:32:49.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:48 smithi007 bash[30436]: audit 2023-12-16T07:32:47.720805+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:49.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:48 smithi007 bash[30436]: audit 2023-12-16T07:32:47.722069+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:32:49.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:48 smithi007 bash[25009]: cluster 2023-12-16T07:32:47.366169+0000 mgr.y (mgr.14152) 146 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:49.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:48 smithi007 bash[25009]: audit 2023-12-16T07:32:47.679398+0000 mon.a (mon.0) 377 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:49.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:48 smithi007 bash[25009]: audit 2023-12-16T07:32:47.719100+0000 mon.a (mon.0) 378 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:32:49.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:48 smithi007 bash[25009]: audit 2023-12-16T07:32:47.720805+0000 mon.a (mon.0) 379 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:49.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:48 smithi007 bash[25009]: audit 2023-12-16T07:32:47.722069+0000 mon.a (mon.0) 380 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:32:50.876 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:50 smithi007 bash[25009]: cluster 2023-12-16T07:32:49.366814+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:50.876 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:50 smithi007 bash[30436]: cluster 2023-12-16T07:32:49.366814+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:51.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:50 smithi169 bash[25083]: cluster 2023-12-16T07:32:49.366814+0000 mgr.y (mgr.14152) 147 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:52.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:52 smithi169 bash[25083]: cluster 2023-12-16T07:32:51.367374+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:52.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:52 smithi169 bash[25083]: audit 2023-12-16T07:32:51.679937+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:52.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:52 smithi169 bash[25083]: audit 2023-12-16T07:32:51.978037+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:53.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:52 smithi007 bash[25009]: cluster 2023-12-16T07:32:51.367374+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:53.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:52 smithi007 bash[25009]: audit 2023-12-16T07:32:51.679937+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:53.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:52 smithi007 bash[25009]: audit 2023-12-16T07:32:51.978037+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:53.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:52 smithi007 bash[30436]: cluster 2023-12-16T07:32:51.367374+0000 mgr.y (mgr.14152) 148 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:53.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:52 smithi007 bash[30436]: audit 2023-12-16T07:32:51.679937+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:53.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:52 smithi007 bash[30436]: audit 2023-12-16T07:32:51.978037+0000 mon.a (mon.0) 382 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:54.692 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 3 on host 'smithi007' 2023-12-16T07:32:54.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:54 smithi169 bash[25083]: cluster 2023-12-16T07:32:53.368008+0000 mgr.y (mgr.14152) 149 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:55.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:54 smithi007 bash[25009]: cluster 2023-12-16T07:32:53.368008+0000 mgr.y (mgr.14152) 149 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:55.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:54 smithi007 bash[30436]: cluster 2023-12-16T07:32:53.368008+0000 mgr.y (mgr.14152) 149 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:55.347 DEBUG:teuthology.orchestra.run.smithi007:osd.3> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.3.service 2023-12-16T07:32:55.350 INFO:tasks.cephadm:Deploying osd.4 on smithi169 with /dev/vg_nvme/lv_4... 2023-12-16T07:32:55.350 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-12-16T07:32:55.478 INFO:journalctl@ceph.osd.3.smithi007.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:32:55.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:55 smithi169 bash[25083]: audit 2023-12-16T07:32:54.691246+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:55.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:55 smithi169 bash[25083]: audit 2023-12-16T07:32:54.702595+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:32:55.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:55 smithi169 bash[25083]: audit 2023-12-16T07:32:54.704216+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:55.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:55 smithi169 bash[25083]: audit 2023-12-16T07:32:54.705349+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:32:55.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:55 smithi169 bash[25083]: audit 2023-12-16T07:32:55.162349+0000 mon.c (mon.1) 7 : audit [INF] from='osd.3 [v2:172.21.15.7:6826/1623735309,v1:172.21.15.7:6827/1623735309]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T07:32:55.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:55 smithi169 bash[25083]: audit 2023-12-16T07:32:55.163123+0000 mon.a (mon.0) 387 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T07:32:56.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:55 smithi007 bash[25009]: audit 2023-12-16T07:32:54.691246+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:56.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:55 smithi007 bash[25009]: audit 2023-12-16T07:32:54.702595+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:32:56.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:55 smithi007 bash[25009]: audit 2023-12-16T07:32:54.704216+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:56.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:55 smithi007 bash[25009]: audit 2023-12-16T07:32:54.705349+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:32:56.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:55 smithi007 bash[25009]: audit 2023-12-16T07:32:55.162349+0000 mon.c (mon.1) 7 : audit [INF] from='osd.3 [v2:172.21.15.7:6826/1623735309,v1:172.21.15.7:6827/1623735309]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T07:32:56.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:55 smithi007 bash[25009]: audit 2023-12-16T07:32:55.163123+0000 mon.a (mon.0) 387 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T07:32:56.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:55 smithi007 bash[30436]: audit 2023-12-16T07:32:54.691246+0000 mon.a (mon.0) 383 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:32:56.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:55 smithi007 bash[30436]: audit 2023-12-16T07:32:54.702595+0000 mon.a (mon.0) 384 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:32:56.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:55 smithi007 bash[30436]: audit 2023-12-16T07:32:54.704216+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:32:56.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:55 smithi007 bash[30436]: audit 2023-12-16T07:32:54.705349+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:32:56.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:55 smithi007 bash[30436]: audit 2023-12-16T07:32:55.162349+0000 mon.c (mon.1) 7 : audit [INF] from='osd.3 [v2:172.21.15.7:6826/1623735309,v1:172.21.15.7:6827/1623735309]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T07:32:56.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:55 smithi007 bash[30436]: audit 2023-12-16T07:32:55.163123+0000 mon.a (mon.0) 387 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T07:32:56.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:56 smithi169 bash[25083]: cluster 2023-12-16T07:32:55.368601+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:56.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:56 smithi169 bash[25083]: audit 2023-12-16T07:32:55.704830+0000 mon.a (mon.0) 388 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-16T07:32:56.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:56 smithi169 bash[25083]: cluster 2023-12-16T07:32:55.704978+0000 mon.a (mon.0) 389 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2023-12-16T07:32:56.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:56 smithi169 bash[25083]: audit 2023-12-16T07:32:55.705202+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:56.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:56 smithi169 bash[25083]: audit 2023-12-16T07:32:55.706365+0000 mon.c (mon.1) 8 : audit [INF] from='osd.3 [v2:172.21.15.7:6826/1623735309,v1:172.21.15.7:6827/1623735309]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:32:56.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:56 smithi169 bash[25083]: audit 2023-12-16T07:32:55.706970+0000 mon.a (mon.0) 391 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:32:57.065 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:32:56 smithi007 bash[42688]: debug 2023-12-16T07:32:56.710+0000 7f0694b16700 -1 osd.3 0 waiting for initial osdmap 2023-12-16T07:32:57.066 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:32:56 smithi007 bash[42688]: debug 2023-12-16T07:32:56.714+0000 7f068dcaa700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:32:57.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:56 smithi007 bash[25009]: cluster 2023-12-16T07:32:55.368601+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:57.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:56 smithi007 bash[25009]: audit 2023-12-16T07:32:55.704830+0000 mon.a (mon.0) 388 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-16T07:32:57.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:56 smithi007 bash[25009]: cluster 2023-12-16T07:32:55.704978+0000 mon.a (mon.0) 389 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2023-12-16T07:32:57.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:56 smithi007 bash[25009]: audit 2023-12-16T07:32:55.705202+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:57.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:56 smithi007 bash[25009]: audit 2023-12-16T07:32:55.706365+0000 mon.c (mon.1) 8 : audit [INF] from='osd.3 [v2:172.21.15.7:6826/1623735309,v1:172.21.15.7:6827/1623735309]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:32:57.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:56 smithi007 bash[25009]: audit 2023-12-16T07:32:55.706970+0000 mon.a (mon.0) 391 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:32:57.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:56 smithi007 bash[30436]: cluster 2023-12-16T07:32:55.368601+0000 mgr.y (mgr.14152) 150 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:57.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:56 smithi007 bash[30436]: audit 2023-12-16T07:32:55.704830+0000 mon.a (mon.0) 388 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-16T07:32:57.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:56 smithi007 bash[30436]: cluster 2023-12-16T07:32:55.704978+0000 mon.a (mon.0) 389 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2023-12-16T07:32:57.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:56 smithi007 bash[30436]: audit 2023-12-16T07:32:55.705202+0000 mon.a (mon.0) 390 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:57.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:56 smithi007 bash[30436]: audit 2023-12-16T07:32:55.706365+0000 mon.c (mon.1) 8 : audit [INF] from='osd.3 [v2:172.21.15.7:6826/1623735309,v1:172.21.15.7:6827/1623735309]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:32:57.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:56 smithi007 bash[30436]: audit 2023-12-16T07:32:55.706970+0000 mon.a (mon.0) 391 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:32:57.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:57 smithi007 bash[25009]: audit 2023-12-16T07:32:56.705446+0000 mon.a (mon.0) 392 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-12-16T07:32:57.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:57 smithi007 bash[25009]: cluster 2023-12-16T07:32:56.705697+0000 mon.a (mon.0) 393 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2023-12-16T07:32:57.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:57 smithi007 bash[25009]: audit 2023-12-16T07:32:56.706533+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:57.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:57 smithi007 bash[25009]: audit 2023-12-16T07:32:56.713923+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:57.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:57 smithi007 bash[30436]: audit 2023-12-16T07:32:56.705446+0000 mon.a (mon.0) 392 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-12-16T07:32:57.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:57 smithi007 bash[30436]: cluster 2023-12-16T07:32:56.705697+0000 mon.a (mon.0) 393 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2023-12-16T07:32:57.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:57 smithi007 bash[30436]: audit 2023-12-16T07:32:56.706533+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:57.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:57 smithi007 bash[30436]: audit 2023-12-16T07:32:56.713923+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:57.972 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:57 smithi169 bash[25083]: audit 2023-12-16T07:32:56.705446+0000 mon.a (mon.0) 392 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2023-12-16T07:32:57.972 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:57 smithi169 bash[25083]: cluster 2023-12-16T07:32:56.705697+0000 mon.a (mon.0) 393 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2023-12-16T07:32:57.972 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:57 smithi169 bash[25083]: audit 2023-12-16T07:32:56.706533+0000 mon.a (mon.0) 394 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:57.972 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:57 smithi169 bash[25083]: audit 2023-12-16T07:32:56.713923+0000 mon.a (mon.0) 395 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:58.504 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-12-16T07:32:58.504 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-12-16T07:32:58.504 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2023-12-16T07:32:58.504 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2023-12-16T07:32:58.504 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.018616 s, 563 MB/s 2023-12-16T07:32:58.505 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2023-12-16T07:32:58.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:58 smithi169 bash[25083]: cluster 2023-12-16T07:32:56.146266+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:32:58.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:58 smithi169 bash[25083]: cluster 2023-12-16T07:32:56.146457+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:32:58.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:58 smithi169 bash[25083]: cluster 2023-12-16T07:32:57.369308+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:58.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:58 smithi169 bash[25083]: cluster 2023-12-16T07:32:57.716534+0000 mon.a (mon.0) 396 : cluster [INF] osd.3 [v2:172.21.15.7:6826/1623735309,v1:172.21.15.7:6827/1623735309] boot 2023-12-16T07:32:58.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:58 smithi169 bash[25083]: cluster 2023-12-16T07:32:57.716703+0000 mon.a (mon.0) 397 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2023-12-16T07:32:58.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:32:58 smithi169 bash[25083]: audit 2023-12-16T07:32:57.717445+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:59.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:58 smithi007 bash[25009]: cluster 2023-12-16T07:32:56.146266+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:32:59.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:58 smithi007 bash[25009]: cluster 2023-12-16T07:32:56.146457+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:32:59.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:58 smithi007 bash[25009]: cluster 2023-12-16T07:32:57.369308+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:59.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:58 smithi007 bash[25009]: cluster 2023-12-16T07:32:57.716534+0000 mon.a (mon.0) 396 : cluster [INF] osd.3 [v2:172.21.15.7:6826/1623735309,v1:172.21.15.7:6827/1623735309] boot 2023-12-16T07:32:59.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:58 smithi007 bash[25009]: cluster 2023-12-16T07:32:57.716703+0000 mon.a (mon.0) 397 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2023-12-16T07:32:59.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:32:58 smithi007 bash[25009]: audit 2023-12-16T07:32:57.717445+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:59.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:58 smithi007 bash[30436]: cluster 2023-12-16T07:32:56.146266+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:32:59.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:58 smithi007 bash[30436]: cluster 2023-12-16T07:32:56.146457+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:32:59.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:58 smithi007 bash[30436]: cluster 2023-12-16T07:32:57.369308+0000 mgr.y (mgr.14152) 151 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2023-12-16T07:32:59.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:58 smithi007 bash[30436]: cluster 2023-12-16T07:32:57.716534+0000 mon.a (mon.0) 396 : cluster [INF] osd.3 [v2:172.21.15.7:6826/1623735309,v1:172.21.15.7:6827/1623735309] boot 2023-12-16T07:32:59.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:58 smithi007 bash[30436]: cluster 2023-12-16T07:32:57.716703+0000 mon.a (mon.0) 397 : cluster [DBG] osdmap e25: 4 total, 4 up, 4 in 2023-12-16T07:32:59.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:32:58 smithi007 bash[30436]: audit 2023-12-16T07:32:57.717445+0000 mon.a (mon.0) 398 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:32:59.146 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_4 2023-12-16T07:33:00.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:00 smithi007 bash[30436]: cluster 2023-12-16T07:32:59.034209+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2023-12-16T07:33:00.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:00 smithi007 bash[25009]: cluster 2023-12-16T07:32:59.034209+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2023-12-16T07:33:00.344 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:00 smithi169 bash[25083]: cluster 2023-12-16T07:32:59.034209+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2023-12-16T07:33:01.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:01 smithi169 bash[25083]: cluster 2023-12-16T07:32:59.369939+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:01.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:01 smithi007 bash[25009]: cluster 2023-12-16T07:32:59.369939+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:01.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:01 smithi007 bash[30436]: cluster 2023-12-16T07:32:59.369939+0000 mgr.y (mgr.14152) 152 : cluster [DBG] pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:02.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:02 smithi169 bash[25083]: audit 2023-12-16T07:33:01.892154+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:33:02.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:02 smithi169 bash[25083]: audit 2023-12-16T07:33:01.898900+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:33:02.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:02 smithi169 bash[25083]: audit 2023-12-16T07:33:01.900675+0000 mon.a (mon.0) 402 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:02.300 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:02 smithi007 bash[30436]: audit 2023-12-16T07:33:01.892154+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:33:02.300 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:02 smithi007 bash[30436]: audit 2023-12-16T07:33:01.898900+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:33:02.300 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:02 smithi007 bash[30436]: audit 2023-12-16T07:33:01.900675+0000 mon.a (mon.0) 402 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:02.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:02 smithi007 bash[25009]: audit 2023-12-16T07:33:01.892154+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:33:02.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:02 smithi007 bash[25009]: audit 2023-12-16T07:33:01.898900+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:33:02.301 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:02 smithi007 bash[25009]: audit 2023-12-16T07:33:01.900675+0000 mon.a (mon.0) 402 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:03.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[25009]: cluster 2023-12-16T07:33:01.370701+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:03.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[25009]: audit 2023-12-16T07:33:01.885565+0000 mgr.y (mgr.14152) 154 : audit [DBG] from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:33:03.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[25009]: audit 2023-12-16T07:33:02.317566+0000 mon.a (mon.0) 403 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:03.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[25009]: audit 2023-12-16T07:33:02.319621+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:03.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[25009]: audit 2023-12-16T07:33:02.320968+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:03.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[25009]: audit 2023-12-16T07:33:02.322182+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:03.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[25009]: audit 2023-12-16T07:33:02.323446+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:03.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[25009]: audit 2023-12-16T07:33:02.330066+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:03.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[25009]: audit 2023-12-16T07:33:02.336775+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:03.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[30436]: cluster 2023-12-16T07:33:01.370701+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:03.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[30436]: audit 2023-12-16T07:33:01.885565+0000 mgr.y (mgr.14152) 154 : audit [DBG] from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:33:03.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[30436]: audit 2023-12-16T07:33:02.317566+0000 mon.a (mon.0) 403 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:03.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[30436]: audit 2023-12-16T07:33:02.319621+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:03.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[30436]: audit 2023-12-16T07:33:02.320968+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:03.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[30436]: audit 2023-12-16T07:33:02.322182+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:03.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[30436]: audit 2023-12-16T07:33:02.323446+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:03.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[30436]: audit 2023-12-16T07:33:02.330066+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:03.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:03 smithi007 bash[30436]: audit 2023-12-16T07:33:02.336775+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:03.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:03 smithi169 bash[25083]: cluster 2023-12-16T07:33:01.370701+0000 mgr.y (mgr.14152) 153 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:03.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:03 smithi169 bash[25083]: audit 2023-12-16T07:33:01.885565+0000 mgr.y (mgr.14152) 154 : audit [DBG] from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:33:03.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:03 smithi169 bash[25083]: audit 2023-12-16T07:33:02.317566+0000 mon.a (mon.0) 403 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:03.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:03 smithi169 bash[25083]: audit 2023-12-16T07:33:02.319621+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:03.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:03 smithi169 bash[25083]: audit 2023-12-16T07:33:02.320968+0000 mon.a (mon.0) 405 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:03.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:03 smithi169 bash[25083]: audit 2023-12-16T07:33:02.322182+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:03.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:03 smithi169 bash[25083]: audit 2023-12-16T07:33:02.323446+0000 mon.a (mon.0) 407 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:03.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:03 smithi169 bash[25083]: audit 2023-12-16T07:33:02.330066+0000 mon.a (mon.0) 408 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:03.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:03 smithi169 bash[25083]: audit 2023-12-16T07:33:02.336775+0000 mon.a (mon.0) 409 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:04.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:04 smithi007 bash[25009]: cephadm 2023-12-16T07:33:02.306502+0000 mgr.y (mgr.14152) 155 : cephadm [INF] Detected new or changed devices on smithi007 2023-12-16T07:33:04.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:04 smithi007 bash[25009]: cephadm 2023-12-16T07:33:02.324381+0000 mgr.y (mgr.14152) 156 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 4062M 2023-12-16T07:33:04.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:04 smithi007 bash[30436]: cephadm 2023-12-16T07:33:02.306502+0000 mgr.y (mgr.14152) 155 : cephadm [INF] Detected new or changed devices on smithi007 2023-12-16T07:33:04.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:04 smithi007 bash[30436]: cephadm 2023-12-16T07:33:02.324381+0000 mgr.y (mgr.14152) 156 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 4062M 2023-12-16T07:33:04.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:04 smithi169 bash[25083]: cephadm 2023-12-16T07:33:02.306502+0000 mgr.y (mgr.14152) 155 : cephadm [INF] Detected new or changed devices on smithi007 2023-12-16T07:33:04.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:04 smithi169 bash[25083]: cephadm 2023-12-16T07:33:02.324381+0000 mgr.y (mgr.14152) 156 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 4062M 2023-12-16T07:33:05.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:05 smithi007 bash[25009]: cluster 2023-12-16T07:33:03.371342+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:05.802 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:05 smithi169 bash[25083]: cluster 2023-12-16T07:33:03.371342+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:05.802 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:05 smithi007 bash[30436]: cluster 2023-12-16T07:33:03.371342+0000 mgr.y (mgr.14152) 157 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:07.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:07 smithi007 bash[25009]: cluster 2023-12-16T07:33:05.371925+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:07.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:07 smithi007 bash[30436]: cluster 2023-12-16T07:33:05.371925+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:07.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:07 smithi169 bash[25083]: cluster 2023-12-16T07:33:05.371925+0000 mgr.y (mgr.14152) 158 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:08.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:08 smithi169 bash[25083]: audit 2023-12-16T07:33:07.934998+0000 mon.b (mon.2) 4 : audit [INF] from='client.? 172.21.15.169:0/1885949598' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5631b50-8179-4ed7-8f03-4de60d19bb69"}]: dispatch 2023-12-16T07:33:08.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:08 smithi169 bash[25083]: audit 2023-12-16T07:33:07.935303+0000 mon.a (mon.0) 410 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5631b50-8179-4ed7-8f03-4de60d19bb69"}]: dispatch 2023-12-16T07:33:08.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:08 smithi169 bash[25083]: audit 2023-12-16T07:33:07.944561+0000 mon.a (mon.0) 411 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5631b50-8179-4ed7-8f03-4de60d19bb69"}]': finished 2023-12-16T07:33:08.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:08 smithi169 bash[25083]: cluster 2023-12-16T07:33:07.944805+0000 mon.a (mon.0) 412 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2023-12-16T07:33:08.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:08 smithi169 bash[25083]: audit 2023-12-16T07:33:07.945084+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:08.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:08 smithi007 bash[25009]: audit 2023-12-16T07:33:07.934998+0000 mon.b (mon.2) 4 : audit [INF] from='client.? 172.21.15.169:0/1885949598' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5631b50-8179-4ed7-8f03-4de60d19bb69"}]: dispatch 2023-12-16T07:33:08.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:08 smithi007 bash[25009]: audit 2023-12-16T07:33:07.935303+0000 mon.a (mon.0) 410 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5631b50-8179-4ed7-8f03-4de60d19bb69"}]: dispatch 2023-12-16T07:33:08.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:08 smithi007 bash[25009]: audit 2023-12-16T07:33:07.944561+0000 mon.a (mon.0) 411 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5631b50-8179-4ed7-8f03-4de60d19bb69"}]': finished 2023-12-16T07:33:08.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:08 smithi007 bash[25009]: cluster 2023-12-16T07:33:07.944805+0000 mon.a (mon.0) 412 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2023-12-16T07:33:08.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:08 smithi007 bash[25009]: audit 2023-12-16T07:33:07.945084+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:08.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:08 smithi007 bash[30436]: audit 2023-12-16T07:33:07.934998+0000 mon.b (mon.2) 4 : audit [INF] from='client.? 172.21.15.169:0/1885949598' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5631b50-8179-4ed7-8f03-4de60d19bb69"}]: dispatch 2023-12-16T07:33:08.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:08 smithi007 bash[30436]: audit 2023-12-16T07:33:07.935303+0000 mon.a (mon.0) 410 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5631b50-8179-4ed7-8f03-4de60d19bb69"}]: dispatch 2023-12-16T07:33:08.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:08 smithi007 bash[30436]: audit 2023-12-16T07:33:07.944561+0000 mon.a (mon.0) 411 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5631b50-8179-4ed7-8f03-4de60d19bb69"}]': finished 2023-12-16T07:33:08.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:08 smithi007 bash[30436]: cluster 2023-12-16T07:33:07.944805+0000 mon.a (mon.0) 412 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2023-12-16T07:33:08.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:08 smithi007 bash[30436]: audit 2023-12-16T07:33:07.945084+0000 mon.a (mon.0) 413 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:09.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:09 smithi169 bash[25083]: cluster 2023-12-16T07:33:07.372546+0000 mgr.y (mgr.14152) 159 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:09.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:09 smithi169 bash[25083]: audit 2023-12-16T07:33:08.631842+0000 mon.b (mon.2) 5 : audit [DBG] from='client.? 172.21.15.169:0/3552719821' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:33:09.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:09 smithi007 bash[25009]: cluster 2023-12-16T07:33:07.372546+0000 mgr.y (mgr.14152) 159 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:09.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:09 smithi007 bash[25009]: audit 2023-12-16T07:33:08.631842+0000 mon.b (mon.2) 5 : audit [DBG] from='client.? 172.21.15.169:0/3552719821' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:33:09.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:09 smithi007 bash[30436]: cluster 2023-12-16T07:33:07.372546+0000 mgr.y (mgr.14152) 159 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:09.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:09 smithi007 bash[30436]: audit 2023-12-16T07:33:08.631842+0000 mon.b (mon.2) 5 : audit [DBG] from='client.? 172.21.15.169:0/3552719821' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:33:10.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:10 smithi169 bash[25083]: cluster 2023-12-16T07:33:09.373132+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:10.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:10 smithi007 bash[25009]: cluster 2023-12-16T07:33:09.373132+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:10.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:10 smithi007 bash[30436]: cluster 2023-12-16T07:33:09.373132+0000 mgr.y (mgr.14152) 160 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:13.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:13 smithi007 bash[25009]: cluster 2023-12-16T07:33:11.373702+0000 mgr.y (mgr.14152) 161 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:13.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:13 smithi007 bash[30436]: cluster 2023-12-16T07:33:11.373702+0000 mgr.y (mgr.14152) 161 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:13.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:13 smithi169 bash[25083]: cluster 2023-12-16T07:33:11.373702+0000 mgr.y (mgr.14152) 161 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:15.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:15 smithi007 bash[25009]: cluster 2023-12-16T07:33:13.374282+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:15.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:15 smithi007 bash[30436]: cluster 2023-12-16T07:33:13.374282+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:15.380 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:15 smithi169 bash[25083]: cluster 2023-12-16T07:33:13.374282+0000 mgr.y (mgr.14152) 162 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:16.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:16 smithi007 bash[25009]: audit 2023-12-16T07:33:15.392157+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:33:16.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:16 smithi007 bash[25009]: audit 2023-12-16T07:33:15.394769+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:33:16.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:16 smithi007 bash[30436]: audit 2023-12-16T07:33:15.392157+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:33:16.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:16 smithi007 bash[30436]: audit 2023-12-16T07:33:15.394769+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:33:16.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:16 smithi169 bash[25083]: audit 2023-12-16T07:33:15.392157+0000 mon.a (mon.0) 414 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:33:16.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:16 smithi169 bash[25083]: audit 2023-12-16T07:33:15.394769+0000 mon.a (mon.0) 415 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:33:17.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:17 smithi007 bash[25009]: cluster 2023-12-16T07:33:15.374984+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:17.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:17 smithi007 bash[30436]: cluster 2023-12-16T07:33:15.374984+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:17.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:17 smithi169 bash[25083]: cluster 2023-12-16T07:33:15.374984+0000 mgr.y (mgr.14152) 163 : cluster [DBG] pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:19.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:19 smithi007 bash[25009]: cluster 2023-12-16T07:33:17.375647+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:19.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:19 smithi007 bash[25009]: audit 2023-12-16T07:33:18.126230+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-16T07:33:19.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:19 smithi007 bash[25009]: audit 2023-12-16T07:33:18.128296+0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:19.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:19 smithi007 bash[30436]: cluster 2023-12-16T07:33:17.375647+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:19.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:19 smithi007 bash[30436]: audit 2023-12-16T07:33:18.126230+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-16T07:33:19.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:19 smithi007 bash[30436]: audit 2023-12-16T07:33:18.128296+0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:19.334 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:19 smithi169 bash[25083]: cluster 2023-12-16T07:33:17.375647+0000 mgr.y (mgr.14152) 164 : cluster [DBG] pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:19.334 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:19 smithi169 bash[25083]: audit 2023-12-16T07:33:18.126230+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-16T07:33:19.334 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:19 smithi169 bash[25083]: audit 2023-12-16T07:33:18.128296+0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:20.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:20 smithi007 bash[25009]: cephadm 2023-12-16T07:33:18.129880+0000 mgr.y (mgr.14152) 165 : cephadm [INF] Deploying daemon osd.4 on smithi169 2023-12-16T07:33:20.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:20 smithi007 bash[30436]: cephadm 2023-12-16T07:33:18.129880+0000 mgr.y (mgr.14152) 165 : cephadm [INF] Deploying daemon osd.4 on smithi169 2023-12-16T07:33:20.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:20 smithi169 bash[25083]: cephadm 2023-12-16T07:33:18.129880+0000 mgr.y (mgr.14152) 165 : cephadm [INF] Deploying daemon osd.4 on smithi169 2023-12-16T07:33:21.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:21 smithi007 bash[25009]: cluster 2023-12-16T07:33:19.376225+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:21.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:21 smithi007 bash[30436]: cluster 2023-12-16T07:33:19.376225+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:21.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:21 smithi169 bash[25083]: cluster 2023-12-16T07:33:19.376225+0000 mgr.y (mgr.14152) 166 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:23.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:22 smithi169 bash[25083]: cluster 2023-12-16T07:33:21.376823+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:23.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:22 smithi169 bash[25083]: audit 2023-12-16T07:33:21.986145+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:23.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:22 smithi169 bash[25083]: audit 2023-12-16T07:33:22.006622+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:33:23.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:22 smithi169 bash[25083]: audit 2023-12-16T07:33:22.009244+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:23.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:22 smithi169 bash[25083]: audit 2023-12-16T07:33:22.010867+0000 mon.a (mon.0) 421 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:33:23.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:22 smithi007 bash[25009]: cluster 2023-12-16T07:33:21.376823+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:23.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:22 smithi007 bash[25009]: audit 2023-12-16T07:33:21.986145+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:23.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:22 smithi007 bash[25009]: audit 2023-12-16T07:33:22.006622+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:33:23.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:22 smithi007 bash[25009]: audit 2023-12-16T07:33:22.009244+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:23.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:22 smithi007 bash[25009]: audit 2023-12-16T07:33:22.010867+0000 mon.a (mon.0) 421 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:33:23.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:22 smithi007 bash[30436]: cluster 2023-12-16T07:33:21.376823+0000 mgr.y (mgr.14152) 167 : cluster [DBG] pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:23.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:22 smithi007 bash[30436]: audit 2023-12-16T07:33:21.986145+0000 mon.a (mon.0) 418 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:23.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:22 smithi007 bash[30436]: audit 2023-12-16T07:33:22.006622+0000 mon.a (mon.0) 419 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:33:23.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:22 smithi007 bash[30436]: audit 2023-12-16T07:33:22.009244+0000 mon.a (mon.0) 420 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:23.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:22 smithi007 bash[30436]: audit 2023-12-16T07:33:22.010867+0000 mon.a (mon.0) 421 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:33:25.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:24 smithi169 bash[25083]: cluster 2023-12-16T07:33:23.377420+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:25.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:24 smithi007 bash[25009]: cluster 2023-12-16T07:33:23.377420+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:25.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:24 smithi007 bash[30436]: cluster 2023-12-16T07:33:23.377420+0000 mgr.y (mgr.14152) 168 : cluster [DBG] pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:26.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:26 smithi169 bash[25083]: cluster 2023-12-16T07:33:25.377957+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:26.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:26 smithi169 bash[25083]: audit 2023-12-16T07:33:25.710242+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:26.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:26 smithi169 bash[25083]: audit 2023-12-16T07:33:26.006785+0000 mon.a (mon.0) 423 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:27.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:26 smithi007 bash[25009]: cluster 2023-12-16T07:33:25.377957+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:27.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:26 smithi007 bash[25009]: audit 2023-12-16T07:33:25.710242+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:27.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:26 smithi007 bash[25009]: audit 2023-12-16T07:33:26.006785+0000 mon.a (mon.0) 423 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:27.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:26 smithi007 bash[30436]: cluster 2023-12-16T07:33:25.377957+0000 mgr.y (mgr.14152) 169 : cluster [DBG] pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:27.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:26 smithi007 bash[30436]: audit 2023-12-16T07:33:25.710242+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:27.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:26 smithi007 bash[30436]: audit 2023-12-16T07:33:26.006785+0000 mon.a (mon.0) 423 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:28.657 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 4 on host 'smithi169' 2023-12-16T07:33:29.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:29 smithi169 bash[25083]: cluster 2023-12-16T07:33:27.378539+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:29.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:29 smithi169 bash[25083]: audit 2023-12-16T07:33:28.655689+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:29.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:29 smithi169 bash[25083]: audit 2023-12-16T07:33:28.682519+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:33:29.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:29 smithi169 bash[25083]: audit 2023-12-16T07:33:28.685371+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:29.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:29 smithi169 bash[25083]: audit 2023-12-16T07:33:28.687297+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:33:29.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:29 smithi007 bash[25009]: cluster 2023-12-16T07:33:27.378539+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:29.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:29 smithi007 bash[25009]: audit 2023-12-16T07:33:28.655689+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:29.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:29 smithi007 bash[25009]: audit 2023-12-16T07:33:28.682519+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:33:29.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:29 smithi007 bash[25009]: audit 2023-12-16T07:33:28.685371+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:29.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:29 smithi007 bash[25009]: audit 2023-12-16T07:33:28.687297+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:33:29.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:29 smithi007 bash[30436]: cluster 2023-12-16T07:33:27.378539+0000 mgr.y (mgr.14152) 170 : cluster [DBG] pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:29.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:29 smithi007 bash[30436]: audit 2023-12-16T07:33:28.655689+0000 mon.a (mon.0) 424 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:29.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:29 smithi007 bash[30436]: audit 2023-12-16T07:33:28.682519+0000 mon.a (mon.0) 425 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:33:29.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:29 smithi007 bash[30436]: audit 2023-12-16T07:33:28.685371+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:29.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:29 smithi007 bash[30436]: audit 2023-12-16T07:33:28.687297+0000 mon.a (mon.0) 427 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:33:29.317 DEBUG:teuthology.orchestra.run.smithi169:osd.4> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.4.service 2023-12-16T07:33:29.320 INFO:tasks.cephadm:Deploying osd.5 on smithi169 with /dev/vg_nvme/lv_3... 2023-12-16T07:33:29.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-12-16T07:33:29.490 INFO:journalctl@ceph.osd.4.smithi169.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:33:30.313 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:30 smithi169 bash[25083]: audit 2023-12-16T07:33:29.123406+0000 mon.b (mon.2) 6 : audit [INF] from='osd.4 [v2:172.21.15.169:6800/3747189467,v1:172.21.15.169:6801/3747189467]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T07:33:30.313 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:30 smithi169 bash[25083]: audit 2023-12-16T07:33:29.123747+0000 mon.a (mon.0) 428 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T07:33:30.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:30 smithi007 bash[25009]: audit 2023-12-16T07:33:29.123406+0000 mon.b (mon.2) 6 : audit [INF] from='osd.4 [v2:172.21.15.169:6800/3747189467,v1:172.21.15.169:6801/3747189467]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T07:33:30.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:30 smithi007 bash[25009]: audit 2023-12-16T07:33:29.123747+0000 mon.a (mon.0) 428 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T07:33:30.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:30 smithi007 bash[30436]: audit 2023-12-16T07:33:29.123406+0000 mon.b (mon.2) 6 : audit [INF] from='osd.4 [v2:172.21.15.169:6800/3747189467,v1:172.21.15.169:6801/3747189467]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T07:33:30.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:30 smithi007 bash[30436]: audit 2023-12-16T07:33:29.123747+0000 mon.a (mon.0) 428 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T07:33:31.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:31 smithi007 bash[30436]: cluster 2023-12-16T07:33:29.379204+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:31.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:31 smithi007 bash[30436]: audit 2023-12-16T07:33:30.054498+0000 mon.a (mon.0) 429 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-16T07:33:31.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:31 smithi007 bash[30436]: cluster 2023-12-16T07:33:30.054756+0000 mon.a (mon.0) 430 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2023-12-16T07:33:31.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:31 smithi007 bash[30436]: audit 2023-12-16T07:33:30.055005+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:31.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:31 smithi007 bash[30436]: audit 2023-12-16T07:33:30.056439+0000 mon.b (mon.2) 7 : audit [INF] from='osd.4 [v2:172.21.15.169:6800/3747189467,v1:172.21.15.169:6801/3747189467]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:33:31.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:31 smithi007 bash[30436]: audit 2023-12-16T07:33:30.056716+0000 mon.a (mon.0) 432 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:33:31.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:31 smithi007 bash[25009]: cluster 2023-12-16T07:33:29.379204+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:31.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:31 smithi007 bash[25009]: audit 2023-12-16T07:33:30.054498+0000 mon.a (mon.0) 429 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-16T07:33:31.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:31 smithi007 bash[25009]: cluster 2023-12-16T07:33:30.054756+0000 mon.a (mon.0) 430 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2023-12-16T07:33:31.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:31 smithi007 bash[25009]: audit 2023-12-16T07:33:30.055005+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:31.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:31 smithi007 bash[25009]: audit 2023-12-16T07:33:30.056439+0000 mon.b (mon.2) 7 : audit [INF] from='osd.4 [v2:172.21.15.169:6800/3747189467,v1:172.21.15.169:6801/3747189467]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:33:31.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:31 smithi007 bash[25009]: audit 2023-12-16T07:33:30.056716+0000 mon.a (mon.0) 432 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:33:31.432 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:33:31 smithi169 bash[28185]: debug 2023-12-16T07:33:31.066+0000 7f90edd8f700 -1 osd.4 0 waiting for initial osdmap 2023-12-16T07:33:31.433 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:33:31 smithi169 bash[28185]: debug 2023-12-16T07:33:31.074+0000 7f90ea72a700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:33:31.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:31 smithi169 bash[25083]: cluster 2023-12-16T07:33:29.379204+0000 mgr.y (mgr.14152) 171 : cluster [DBG] pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:31.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:31 smithi169 bash[25083]: audit 2023-12-16T07:33:30.054498+0000 mon.a (mon.0) 429 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-16T07:33:31.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:31 smithi169 bash[25083]: cluster 2023-12-16T07:33:30.054756+0000 mon.a (mon.0) 430 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2023-12-16T07:33:31.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:31 smithi169 bash[25083]: audit 2023-12-16T07:33:30.055005+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:31.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:31 smithi169 bash[25083]: audit 2023-12-16T07:33:30.056439+0000 mon.b (mon.2) 7 : audit [INF] from='osd.4 [v2:172.21.15.169:6800/3747189467,v1:172.21.15.169:6801/3747189467]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:33:31.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:31 smithi169 bash[25083]: audit 2023-12-16T07:33:30.056716+0000 mon.a (mon.0) 432 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:33:32.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:32 smithi007 bash[25009]: audit 2023-12-16T07:33:31.058247+0000 mon.a (mon.0) 433 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T07:33:32.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:32 smithi007 bash[25009]: cluster 2023-12-16T07:33:31.058530+0000 mon.a (mon.0) 434 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2023-12-16T07:33:32.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:32 smithi007 bash[25009]: audit 2023-12-16T07:33:31.059246+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:32.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:32 smithi007 bash[25009]: audit 2023-12-16T07:33:31.066396+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:32.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:32 smithi007 bash[25009]: cluster 2023-12-16T07:33:32.042706+0000 mon.a (mon.0) 437 : cluster [INF] osd.4 [v2:172.21.15.169:6800/3747189467,v1:172.21.15.169:6801/3747189467] boot 2023-12-16T07:33:32.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:32 smithi007 bash[25009]: cluster 2023-12-16T07:33:32.042794+0000 mon.a (mon.0) 438 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2023-12-16T07:33:32.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:32 smithi007 bash[25009]: audit 2023-12-16T07:33:32.043484+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:32.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:32 smithi007 bash[30436]: audit 2023-12-16T07:33:31.058247+0000 mon.a (mon.0) 433 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T07:33:32.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:32 smithi007 bash[30436]: cluster 2023-12-16T07:33:31.058530+0000 mon.a (mon.0) 434 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2023-12-16T07:33:32.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:32 smithi007 bash[30436]: audit 2023-12-16T07:33:31.059246+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:32.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:32 smithi007 bash[30436]: audit 2023-12-16T07:33:31.066396+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:32.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:32 smithi007 bash[30436]: cluster 2023-12-16T07:33:32.042706+0000 mon.a (mon.0) 437 : cluster [INF] osd.4 [v2:172.21.15.169:6800/3747189467,v1:172.21.15.169:6801/3747189467] boot 2023-12-16T07:33:32.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:32 smithi007 bash[30436]: cluster 2023-12-16T07:33:32.042794+0000 mon.a (mon.0) 438 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2023-12-16T07:33:32.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:32 smithi007 bash[30436]: audit 2023-12-16T07:33:32.043484+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:32.320 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:32 smithi169 bash[25083]: audit 2023-12-16T07:33:31.058247+0000 mon.a (mon.0) 433 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T07:33:32.320 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:32 smithi169 bash[25083]: cluster 2023-12-16T07:33:31.058530+0000 mon.a (mon.0) 434 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2023-12-16T07:33:32.320 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:32 smithi169 bash[25083]: audit 2023-12-16T07:33:31.059246+0000 mon.a (mon.0) 435 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:32.320 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:32 smithi169 bash[25083]: audit 2023-12-16T07:33:31.066396+0000 mon.a (mon.0) 436 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:32.320 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:32 smithi169 bash[25083]: cluster 2023-12-16T07:33:32.042706+0000 mon.a (mon.0) 437 : cluster [INF] osd.4 [v2:172.21.15.169:6800/3747189467,v1:172.21.15.169:6801/3747189467] boot 2023-12-16T07:33:32.320 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:32 smithi169 bash[25083]: cluster 2023-12-16T07:33:32.042794+0000 mon.a (mon.0) 438 : cluster [DBG] osdmap e30: 5 total, 5 up, 5 in 2023-12-16T07:33:32.320 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:32 smithi169 bash[25083]: audit 2023-12-16T07:33:32.043484+0000 mon.a (mon.0) 439 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:33:32.912 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-12-16T07:33:32.912 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-12-16T07:33:32.912 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2023-12-16T07:33:32.912 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2023-12-16T07:33:32.912 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0196754 s, 533 MB/s 2023-12-16T07:33:32.912 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2023-12-16T07:33:33.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:33 smithi169 bash[25083]: cluster 2023-12-16T07:33:30.154515+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:33:33.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:33 smithi169 bash[25083]: cluster 2023-12-16T07:33:30.154697+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:33:33.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:33 smithi169 bash[25083]: cluster 2023-12-16T07:33:31.379823+0000 mgr.y (mgr.14152) 172 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:33.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:33 smithi169 bash[25083]: cluster 2023-12-16T07:33:33.052951+0000 mon.a (mon.0) 440 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2023-12-16T07:33:33.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:33 smithi007 bash[25009]: cluster 2023-12-16T07:33:30.154515+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:33:33.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:33 smithi007 bash[25009]: cluster 2023-12-16T07:33:30.154697+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:33:33.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:33 smithi007 bash[25009]: cluster 2023-12-16T07:33:31.379823+0000 mgr.y (mgr.14152) 172 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:33.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:33 smithi007 bash[25009]: cluster 2023-12-16T07:33:33.052951+0000 mon.a (mon.0) 440 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2023-12-16T07:33:33.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:33 smithi007 bash[30436]: cluster 2023-12-16T07:33:30.154515+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:33:33.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:33 smithi007 bash[30436]: cluster 2023-12-16T07:33:30.154697+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:33:33.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:33 smithi007 bash[30436]: cluster 2023-12-16T07:33:31.379823+0000 mgr.y (mgr.14152) 172 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-12-16T07:33:33.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:33 smithi007 bash[30436]: cluster 2023-12-16T07:33:33.052951+0000 mon.a (mon.0) 440 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2023-12-16T07:33:34.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_3 2023-12-16T07:33:35.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:35 smithi169 bash[25083]: cluster 2023-12-16T07:33:33.380404+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v150: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:35.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:35 smithi169 bash[25083]: cluster 2023-12-16T07:33:34.054756+0000 mon.a (mon.0) 441 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2023-12-16T07:33:35.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:35 smithi007 bash[25009]: cluster 2023-12-16T07:33:33.380404+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v150: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:35.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:35 smithi007 bash[25009]: cluster 2023-12-16T07:33:34.054756+0000 mon.a (mon.0) 441 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2023-12-16T07:33:35.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:35 smithi007 bash[30436]: cluster 2023-12-16T07:33:33.380404+0000 mgr.y (mgr.14152) 173 : cluster [DBG] pgmap v150: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:35.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:35 smithi007 bash[30436]: cluster 2023-12-16T07:33:34.054756+0000 mon.a (mon.0) 441 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2023-12-16T07:33:36.376 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:36 smithi169 bash[25083]: cluster 2023-12-16T07:33:35.057451+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2023-12-16T07:33:36.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:36 smithi007 bash[25009]: cluster 2023-12-16T07:33:35.057451+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2023-12-16T07:33:36.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:36 smithi007 bash[30436]: cluster 2023-12-16T07:33:35.057451+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2023-12-16T07:33:37.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:37 smithi169 bash[25083]: cluster 2023-12-16T07:33:35.380978+0000 mgr.y (mgr.14152) 174 : cluster [DBG] pgmap v153: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:37.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:37 smithi169 bash[25083]: cluster 2023-12-16T07:33:36.067908+0000 mon.a (mon.0) 443 : cluster [DBG] osdmap e34: 5 total, 5 up, 5 in 2023-12-16T07:33:37.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:37 smithi007 bash[25009]: cluster 2023-12-16T07:33:35.380978+0000 mgr.y (mgr.14152) 174 : cluster [DBG] pgmap v153: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:37.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:37 smithi007 bash[25009]: cluster 2023-12-16T07:33:36.067908+0000 mon.a (mon.0) 443 : cluster [DBG] osdmap e34: 5 total, 5 up, 5 in 2023-12-16T07:33:37.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:37 smithi007 bash[30436]: cluster 2023-12-16T07:33:35.380978+0000 mgr.y (mgr.14152) 174 : cluster [DBG] pgmap v153: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:37.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:37 smithi007 bash[30436]: cluster 2023-12-16T07:33:36.067908+0000 mon.a (mon.0) 443 : cluster [DBG] osdmap e34: 5 total, 5 up, 5 in 2023-12-16T07:33:38.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[25009]: cluster 2023-12-16T07:33:37.381629+0000 mgr.y (mgr.14152) 175 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 119 KiB/s, 0 objects/s recovering 2023-12-16T07:33:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[25009]: cephadm 2023-12-16T07:33:37.477638+0000 mgr.y (mgr.14152) 176 : cephadm [INF] Detected new or changed devices on smithi169 2023-12-16T07:33:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[25009]: audit 2023-12-16T07:33:37.487856+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[25009]: audit 2023-12-16T07:33:37.489523+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[25009]: cephadm 2023-12-16T07:33:37.490757+0000 mgr.y (mgr.14152) 177 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 17271M 2023-12-16T07:33:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[25009]: audit 2023-12-16T07:33:37.498028+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[25009]: audit 2023-12-16T07:33:37.504159+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[25009]: audit 2023-12-16T07:33:37.948390+0000 mgr.y (mgr.14152) 178 : audit [DBG] from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:33:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[25009]: audit 2023-12-16T07:33:37.955243+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:33:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[25009]: audit 2023-12-16T07:33:37.961942+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:33:38.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[25009]: audit 2023-12-16T07:33:37.964058+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:38.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[30436]: cluster 2023-12-16T07:33:37.381629+0000 mgr.y (mgr.14152) 175 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 119 KiB/s, 0 objects/s recovering 2023-12-16T07:33:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[30436]: cephadm 2023-12-16T07:33:37.477638+0000 mgr.y (mgr.14152) 176 : cephadm [INF] Detected new or changed devices on smithi169 2023-12-16T07:33:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[30436]: audit 2023-12-16T07:33:37.487856+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[30436]: audit 2023-12-16T07:33:37.489523+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[30436]: cephadm 2023-12-16T07:33:37.490757+0000 mgr.y (mgr.14152) 177 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 17271M 2023-12-16T07:33:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[30436]: audit 2023-12-16T07:33:37.498028+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[30436]: audit 2023-12-16T07:33:37.504159+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[30436]: audit 2023-12-16T07:33:37.948390+0000 mgr.y (mgr.14152) 178 : audit [DBG] from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:33:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[30436]: audit 2023-12-16T07:33:37.955243+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:33:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[30436]: audit 2023-12-16T07:33:37.961942+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:33:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:38 smithi007 bash[30436]: audit 2023-12-16T07:33:37.964058+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:38.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:38 smithi169 bash[25083]: cluster 2023-12-16T07:33:37.381629+0000 mgr.y (mgr.14152) 175 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 119 KiB/s, 0 objects/s recovering 2023-12-16T07:33:38.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:38 smithi169 bash[25083]: cephadm 2023-12-16T07:33:37.477638+0000 mgr.y (mgr.14152) 176 : cephadm [INF] Detected new or changed devices on smithi169 2023-12-16T07:33:38.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:38 smithi169 bash[25083]: audit 2023-12-16T07:33:37.487856+0000 mon.a (mon.0) 444 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:38.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:38 smithi169 bash[25083]: audit 2023-12-16T07:33:37.489523+0000 mon.a (mon.0) 445 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:33:38.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:38 smithi169 bash[25083]: cephadm 2023-12-16T07:33:37.490757+0000 mgr.y (mgr.14152) 177 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 17271M 2023-12-16T07:33:38.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:38 smithi169 bash[25083]: audit 2023-12-16T07:33:37.498028+0000 mon.a (mon.0) 446 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:38.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:38 smithi169 bash[25083]: audit 2023-12-16T07:33:37.504159+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:38.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:38 smithi169 bash[25083]: audit 2023-12-16T07:33:37.948390+0000 mgr.y (mgr.14152) 178 : audit [DBG] from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:33:38.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:38 smithi169 bash[25083]: audit 2023-12-16T07:33:37.955243+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:33:38.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:38 smithi169 bash[25083]: audit 2023-12-16T07:33:37.961942+0000 mon.a (mon.0) 449 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:33:38.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:38 smithi169 bash[25083]: audit 2023-12-16T07:33:37.964058+0000 mon.a (mon.0) 450 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:40.815 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:40 smithi007 bash[30436]: cluster 2023-12-16T07:33:39.382238+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2023-12-16T07:33:40.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:40 smithi007 bash[25009]: cluster 2023-12-16T07:33:39.382238+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2023-12-16T07:33:40.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:40 smithi169 bash[25083]: cluster 2023-12-16T07:33:39.382238+0000 mgr.y (mgr.14152) 179 : cluster [DBG] pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2023-12-16T07:33:43.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:43 smithi007 bash[25009]: cluster 2023-12-16T07:33:41.382865+0000 mgr.y (mgr.14152) 180 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-12-16T07:33:43.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:43 smithi007 bash[30436]: cluster 2023-12-16T07:33:41.382865+0000 mgr.y (mgr.14152) 180 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-12-16T07:33:43.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:43 smithi169 bash[25083]: cluster 2023-12-16T07:33:41.382865+0000 mgr.y (mgr.14152) 180 : cluster [DBG] pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-12-16T07:33:44.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:44 smithi007 bash[25009]: audit 2023-12-16T07:33:44.028266+0000 mon.b (mon.2) 8 : audit [INF] from='client.? 172.21.15.169:0/4275806626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70f322c0-b8b5-439c-bf2a-8688a28ca919"}]: dispatch 2023-12-16T07:33:44.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:44 smithi007 bash[25009]: audit 2023-12-16T07:33:44.028736+0000 mon.a (mon.0) 451 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70f322c0-b8b5-439c-bf2a-8688a28ca919"}]: dispatch 2023-12-16T07:33:44.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:44 smithi007 bash[25009]: audit 2023-12-16T07:33:44.038093+0000 mon.a (mon.0) 452 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70f322c0-b8b5-439c-bf2a-8688a28ca919"}]': finished 2023-12-16T07:33:44.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:44 smithi007 bash[25009]: cluster 2023-12-16T07:33:44.038309+0000 mon.a (mon.0) 453 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2023-12-16T07:33:44.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:44 smithi007 bash[25009]: audit 2023-12-16T07:33:44.038694+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:33:44.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:44 smithi007 bash[30436]: audit 2023-12-16T07:33:44.028266+0000 mon.b (mon.2) 8 : audit [INF] from='client.? 172.21.15.169:0/4275806626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70f322c0-b8b5-439c-bf2a-8688a28ca919"}]: dispatch 2023-12-16T07:33:44.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:44 smithi007 bash[30436]: audit 2023-12-16T07:33:44.028736+0000 mon.a (mon.0) 451 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70f322c0-b8b5-439c-bf2a-8688a28ca919"}]: dispatch 2023-12-16T07:33:44.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:44 smithi007 bash[30436]: audit 2023-12-16T07:33:44.038093+0000 mon.a (mon.0) 452 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70f322c0-b8b5-439c-bf2a-8688a28ca919"}]': finished 2023-12-16T07:33:44.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:44 smithi007 bash[30436]: cluster 2023-12-16T07:33:44.038309+0000 mon.a (mon.0) 453 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2023-12-16T07:33:44.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:44 smithi007 bash[30436]: audit 2023-12-16T07:33:44.038694+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:33:44.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:44 smithi169 bash[25083]: audit 2023-12-16T07:33:44.028266+0000 mon.b (mon.2) 8 : audit [INF] from='client.? 172.21.15.169:0/4275806626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70f322c0-b8b5-439c-bf2a-8688a28ca919"}]: dispatch 2023-12-16T07:33:44.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:44 smithi169 bash[25083]: audit 2023-12-16T07:33:44.028736+0000 mon.a (mon.0) 451 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70f322c0-b8b5-439c-bf2a-8688a28ca919"}]: dispatch 2023-12-16T07:33:44.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:44 smithi169 bash[25083]: audit 2023-12-16T07:33:44.038093+0000 mon.a (mon.0) 452 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70f322c0-b8b5-439c-bf2a-8688a28ca919"}]': finished 2023-12-16T07:33:44.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:44 smithi169 bash[25083]: cluster 2023-12-16T07:33:44.038309+0000 mon.a (mon.0) 453 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2023-12-16T07:33:44.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:44 smithi169 bash[25083]: audit 2023-12-16T07:33:44.038694+0000 mon.a (mon.0) 454 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:33:45.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:45 smithi007 bash[25009]: cluster 2023-12-16T07:33:43.383471+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-12-16T07:33:45.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:45 smithi007 bash[25009]: audit 2023-12-16T07:33:44.751212+0000 mon.b (mon.2) 9 : audit [DBG] from='client.? 172.21.15.169:0/1629307537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:33:45.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:45 smithi007 bash[30436]: cluster 2023-12-16T07:33:43.383471+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-12-16T07:33:45.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:45 smithi007 bash[30436]: audit 2023-12-16T07:33:44.751212+0000 mon.b (mon.2) 9 : audit [DBG] from='client.? 172.21.15.169:0/1629307537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:33:45.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:45 smithi169 bash[25083]: cluster 2023-12-16T07:33:43.383471+0000 mgr.y (mgr.14152) 181 : cluster [DBG] pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-12-16T07:33:45.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:45 smithi169 bash[25083]: audit 2023-12-16T07:33:44.751212+0000 mon.b (mon.2) 9 : audit [DBG] from='client.? 172.21.15.169:0/1629307537' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:33:47.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:47 smithi007 bash[25009]: cluster 2023-12-16T07:33:45.384097+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 55 KiB/s, 0 objects/s recovering 2023-12-16T07:33:47.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:47 smithi007 bash[30436]: cluster 2023-12-16T07:33:45.384097+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 55 KiB/s, 0 objects/s recovering 2023-12-16T07:33:47.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:47 smithi169 bash[25083]: cluster 2023-12-16T07:33:45.384097+0000 mgr.y (mgr.14152) 182 : cluster [DBG] pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 55 KiB/s, 0 objects/s recovering 2023-12-16T07:33:49.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:49 smithi007 bash[25009]: cluster 2023-12-16T07:33:47.384657+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:49.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:49 smithi007 bash[30436]: cluster 2023-12-16T07:33:47.384657+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:49.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:49 smithi169 bash[25083]: cluster 2023-12-16T07:33:47.384657+0000 mgr.y (mgr.14152) 183 : cluster [DBG] pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:51.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:51 smithi007 bash[25009]: cluster 2023-12-16T07:33:49.385237+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:51.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:51 smithi007 bash[30436]: cluster 2023-12-16T07:33:49.385237+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:51.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:51 smithi169 bash[25083]: cluster 2023-12-16T07:33:49.385237+0000 mgr.y (mgr.14152) 184 : cluster [DBG] pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:53.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:53 smithi169 bash[25083]: cluster 2023-12-16T07:33:51.385844+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:53.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:53 smithi007 bash[25009]: cluster 2023-12-16T07:33:51.385844+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:53.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:53 smithi007 bash[30436]: cluster 2023-12-16T07:33:51.385844+0000 mgr.y (mgr.14152) 185 : cluster [DBG] pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:55.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:55 smithi169 bash[25083]: cluster 2023-12-16T07:33:53.386461+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:55.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:55 smithi169 bash[25083]: audit 2023-12-16T07:33:54.508371+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-16T07:33:55.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:55 smithi169 bash[25083]: audit 2023-12-16T07:33:54.510162+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:55.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:55 smithi007 bash[30436]: cluster 2023-12-16T07:33:53.386461+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:55.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:55 smithi007 bash[30436]: audit 2023-12-16T07:33:54.508371+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-16T07:33:55.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:55 smithi007 bash[30436]: audit 2023-12-16T07:33:54.510162+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:55.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:55 smithi007 bash[25009]: cluster 2023-12-16T07:33:53.386461+0000 mgr.y (mgr.14152) 186 : cluster [DBG] pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:55.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:55 smithi007 bash[25009]: audit 2023-12-16T07:33:54.508371+0000 mon.a (mon.0) 455 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-16T07:33:55.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:55 smithi007 bash[25009]: audit 2023-12-16T07:33:54.510162+0000 mon.a (mon.0) 456 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:56.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:56 smithi169 bash[25083]: cephadm 2023-12-16T07:33:54.511741+0000 mgr.y (mgr.14152) 187 : cephadm [INF] Deploying daemon osd.5 on smithi169 2023-12-16T07:33:56.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:56 smithi007 bash[30436]: cephadm 2023-12-16T07:33:54.511741+0000 mgr.y (mgr.14152) 187 : cephadm [INF] Deploying daemon osd.5 on smithi169 2023-12-16T07:33:56.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:56 smithi007 bash[25009]: cephadm 2023-12-16T07:33:54.511741+0000 mgr.y (mgr.14152) 187 : cephadm [INF] Deploying daemon osd.5 on smithi169 2023-12-16T07:33:57.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:57 smithi169 bash[25083]: cluster 2023-12-16T07:33:55.387048+0000 mgr.y (mgr.14152) 188 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:57.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:57 smithi007 bash[30436]: cluster 2023-12-16T07:33:55.387048+0000 mgr.y (mgr.14152) 188 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:57.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:57 smithi007 bash[25009]: cluster 2023-12-16T07:33:55.387048+0000 mgr.y (mgr.14152) 188 : cluster [DBG] pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:58.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:58 smithi169 bash[25083]: cluster 2023-12-16T07:33:57.387747+0000 mgr.y (mgr.14152) 189 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:58.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:58 smithi007 bash[25009]: cluster 2023-12-16T07:33:57.387747+0000 mgr.y (mgr.14152) 189 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:58.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:58 smithi007 bash[30436]: cluster 2023-12-16T07:33:57.387747+0000 mgr.y (mgr.14152) 189 : cluster [DBG] pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:33:59.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:59 smithi169 bash[25083]: audit 2023-12-16T07:33:58.693924+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:33:59.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:59 smithi169 bash[25083]: audit 2023-12-16T07:33:58.730849+0000 mon.a (mon.0) 458 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:33:59.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:59 smithi169 bash[25083]: audit 2023-12-16T07:33:58.732963+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:33:59.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:33:59 smithi169 bash[25083]: audit 2023-12-16T07:33:58.734250+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:34:00.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:59 smithi007 bash[25009]: audit 2023-12-16T07:33:58.693924+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:00.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:59 smithi007 bash[25009]: audit 2023-12-16T07:33:58.730849+0000 mon.a (mon.0) 458 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:34:00.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:59 smithi007 bash[25009]: audit 2023-12-16T07:33:58.732963+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:00.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:33:59 smithi007 bash[25009]: audit 2023-12-16T07:33:58.734250+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:34:00.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:59 smithi007 bash[30436]: audit 2023-12-16T07:33:58.693924+0000 mon.a (mon.0) 457 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:00.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:59 smithi007 bash[30436]: audit 2023-12-16T07:33:58.730849+0000 mon.a (mon.0) 458 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:34:00.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:59 smithi007 bash[30436]: audit 2023-12-16T07:33:58.732963+0000 mon.a (mon.0) 459 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:00.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:33:59 smithi007 bash[30436]: audit 2023-12-16T07:33:58.734250+0000 mon.a (mon.0) 460 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:34:00.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:00 smithi169 bash[25083]: cluster 2023-12-16T07:33:59.388497+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:01.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:00 smithi007 bash[25009]: cluster 2023-12-16T07:33:59.388497+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:01.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:00 smithi007 bash[30436]: cluster 2023-12-16T07:33:59.388497+0000 mgr.y (mgr.14152) 190 : cluster [DBG] pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:03.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:03 smithi169 bash[25083]: cluster 2023-12-16T07:34:01.389208+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:03.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:03 smithi169 bash[25083]: audit 2023-12-16T07:34:02.498755+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:03.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:03 smithi169 bash[25083]: audit 2023-12-16T07:34:02.506372+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:03.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:03 smithi007 bash[25009]: cluster 2023-12-16T07:34:01.389208+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:03.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:03 smithi007 bash[25009]: audit 2023-12-16T07:34:02.498755+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:03.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:03 smithi007 bash[25009]: audit 2023-12-16T07:34:02.506372+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:03.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:03 smithi007 bash[30436]: cluster 2023-12-16T07:34:01.389208+0000 mgr.y (mgr.14152) 191 : cluster [DBG] pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:03.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:03 smithi007 bash[30436]: audit 2023-12-16T07:34:02.498755+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:03.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:03 smithi007 bash[30436]: audit 2023-12-16T07:34:02.506372+0000 mon.a (mon.0) 462 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:05.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:05 smithi169 bash[25083]: cluster 2023-12-16T07:34:03.389946+0000 mgr.y (mgr.14152) 192 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:05.281 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 5 on host 'smithi169' 2023-12-16T07:34:05.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:05 smithi007 bash[30436]: cluster 2023-12-16T07:34:03.389946+0000 mgr.y (mgr.14152) 192 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:05.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:05 smithi007 bash[25009]: cluster 2023-12-16T07:34:03.389946+0000 mgr.y (mgr.14152) 192 : cluster [DBG] pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:05.915 DEBUG:teuthology.orchestra.run.smithi169:osd.5> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.5.service 2023-12-16T07:34:05.919 INFO:tasks.cephadm:Deploying osd.6 on smithi169 with /dev/vg_nvme/lv_2... 2023-12-16T07:34:05.919 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-12-16T07:34:06.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:06 smithi169 bash[25083]: audit 2023-12-16T07:34:05.279283+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:06.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:06 smithi169 bash[25083]: audit 2023-12-16T07:34:05.281139+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:34:06.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:06 smithi169 bash[25083]: audit 2023-12-16T07:34:05.284627+0000 mon.a (mon.0) 465 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:06.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:06 smithi169 bash[25083]: audit 2023-12-16T07:34:05.286097+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:34:06.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:06 smithi169 bash[25083]: cluster 2023-12-16T07:34:05.390543+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:06.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:06 smithi169 bash[25083]: audit 2023-12-16T07:34:05.966916+0000 mon.b (mon.2) 10 : audit [INF] from='osd.5 [v2:172.21.15.169:6808/3877143549,v1:172.21.15.169:6809/3877143549]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T07:34:06.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:06 smithi169 bash[25083]: audit 2023-12-16T07:34:05.967265+0000 mon.a (mon.0) 467 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T07:34:06.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:06 smithi007 bash[25009]: audit 2023-12-16T07:34:05.279283+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:06.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:06 smithi007 bash[25009]: audit 2023-12-16T07:34:05.281139+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:34:06.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:06 smithi007 bash[25009]: audit 2023-12-16T07:34:05.284627+0000 mon.a (mon.0) 465 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:06.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:06 smithi007 bash[25009]: audit 2023-12-16T07:34:05.286097+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:34:06.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:06 smithi007 bash[25009]: cluster 2023-12-16T07:34:05.390543+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:06.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:06 smithi007 bash[25009]: audit 2023-12-16T07:34:05.966916+0000 mon.b (mon.2) 10 : audit [INF] from='osd.5 [v2:172.21.15.169:6808/3877143549,v1:172.21.15.169:6809/3877143549]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T07:34:06.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:06 smithi007 bash[25009]: audit 2023-12-16T07:34:05.967265+0000 mon.a (mon.0) 467 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T07:34:06.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:06 smithi007 bash[30436]: audit 2023-12-16T07:34:05.279283+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:06.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:06 smithi007 bash[30436]: audit 2023-12-16T07:34:05.281139+0000 mon.a (mon.0) 464 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:34:06.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:06 smithi007 bash[30436]: audit 2023-12-16T07:34:05.284627+0000 mon.a (mon.0) 465 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:06.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:06 smithi007 bash[30436]: audit 2023-12-16T07:34:05.286097+0000 mon.a (mon.0) 466 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:34:06.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:06 smithi007 bash[30436]: cluster 2023-12-16T07:34:05.390543+0000 mgr.y (mgr.14152) 193 : cluster [DBG] pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:06.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:06 smithi007 bash[30436]: audit 2023-12-16T07:34:05.966916+0000 mon.b (mon.2) 10 : audit [INF] from='osd.5 [v2:172.21.15.169:6808/3877143549,v1:172.21.15.169:6809/3877143549]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T07:34:06.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:06 smithi007 bash[30436]: audit 2023-12-16T07:34:05.967265+0000 mon.a (mon.0) 467 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T07:34:06.630 INFO:journalctl@ceph.osd.5.smithi169.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:34:06.630 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:34:05 smithi169 bash[31281]: debug 2023-12-16T07:34:05.961+0000 7fbd153663c0 -1 osd.5 0 log_to_monitors true 2023-12-16T07:34:07.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:07 smithi007 bash[25009]: audit 2023-12-16T07:34:06.290513+0000 mon.a (mon.0) 468 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-16T07:34:08.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:07 smithi007 bash[25009]: cluster 2023-12-16T07:34:06.290734+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2023-12-16T07:34:08.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:07 smithi007 bash[25009]: audit 2023-12-16T07:34:06.291106+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:34:08.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:07 smithi007 bash[25009]: audit 2023-12-16T07:34:06.296004+0000 mon.b (mon.2) 11 : audit [INF] from='osd.5 [v2:172.21.15.169:6808/3877143549,v1:172.21.15.169:6809/3877143549]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:34:08.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:07 smithi007 bash[25009]: audit 2023-12-16T07:34:06.296409+0000 mon.a (mon.0) 471 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:34:08.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:07 smithi007 bash[30436]: audit 2023-12-16T07:34:06.290513+0000 mon.a (mon.0) 468 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-16T07:34:08.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:07 smithi007 bash[30436]: cluster 2023-12-16T07:34:06.290734+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2023-12-16T07:34:08.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:07 smithi007 bash[30436]: audit 2023-12-16T07:34:06.291106+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:34:08.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:07 smithi007 bash[30436]: audit 2023-12-16T07:34:06.296004+0000 mon.b (mon.2) 11 : audit [INF] from='osd.5 [v2:172.21.15.169:6808/3877143549,v1:172.21.15.169:6809/3877143549]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:34:08.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:07 smithi007 bash[30436]: audit 2023-12-16T07:34:06.296409+0000 mon.a (mon.0) 471 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:34:08.074 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:34:07 smithi169 bash[31281]: debug 2023-12-16T07:34:07.305+0000 7fbd0bd69700 -1 osd.5 0 waiting for initial osdmap 2023-12-16T07:34:08.074 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:34:07 smithi169 bash[31281]: debug 2023-12-16T07:34:07.313+0000 7fbd05eff700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:34:08.075 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:07 smithi169 bash[25083]: audit 2023-12-16T07:34:06.290513+0000 mon.a (mon.0) 468 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-16T07:34:08.075 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:07 smithi169 bash[25083]: cluster 2023-12-16T07:34:06.290734+0000 mon.a (mon.0) 469 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2023-12-16T07:34:08.075 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:07 smithi169 bash[25083]: audit 2023-12-16T07:34:06.291106+0000 mon.a (mon.0) 470 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:34:08.075 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:07 smithi169 bash[25083]: audit 2023-12-16T07:34:06.296004+0000 mon.b (mon.2) 11 : audit [INF] from='osd.5 [v2:172.21.15.169:6808/3877143549,v1:172.21.15.169:6809/3877143549]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:34:08.075 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:07 smithi169 bash[25083]: audit 2023-12-16T07:34:06.296409+0000 mon.a (mon.0) 471 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:34:08.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[25009]: audit 2023-12-16T07:34:07.292187+0000 mon.a (mon.0) 472 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T07:34:08.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[25009]: cluster 2023-12-16T07:34:07.292558+0000 mon.a (mon.0) 473 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2023-12-16T07:34:08.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[25009]: audit 2023-12-16T07:34:07.293431+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:34:08.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[25009]: audit 2023-12-16T07:34:07.305083+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:34:08.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[25009]: cluster 2023-12-16T07:34:07.391176+0000 mgr.y (mgr.14152) 194 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:08.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[25009]: cluster 2023-12-16T07:34:08.296105+0000 mon.a (mon.0) 476 : cluster [INF] osd.5 [v2:172.21.15.169:6808/3877143549,v1:172.21.15.169:6809/3877143549] boot 2023-12-16T07:34:08.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[25009]: cluster 2023-12-16T07:34:08.296252+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2023-12-16T07:34:08.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[25009]: audit 2023-12-16T07:34:08.296605+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:34:08.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[30436]: audit 2023-12-16T07:34:07.292187+0000 mon.a (mon.0) 472 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T07:34:08.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[30436]: cluster 2023-12-16T07:34:07.292558+0000 mon.a (mon.0) 473 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2023-12-16T07:34:08.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[30436]: audit 2023-12-16T07:34:07.293431+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:34:08.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[30436]: audit 2023-12-16T07:34:07.305083+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:34:08.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[30436]: cluster 2023-12-16T07:34:07.391176+0000 mgr.y (mgr.14152) 194 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:08.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[30436]: cluster 2023-12-16T07:34:08.296105+0000 mon.a (mon.0) 476 : cluster [INF] osd.5 [v2:172.21.15.169:6808/3877143549,v1:172.21.15.169:6809/3877143549] boot 2023-12-16T07:34:08.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[30436]: cluster 2023-12-16T07:34:08.296252+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2023-12-16T07:34:08.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:08 smithi007 bash[30436]: audit 2023-12-16T07:34:08.296605+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:34:08.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:08 smithi169 bash[25083]: audit 2023-12-16T07:34:07.292187+0000 mon.a (mon.0) 472 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T07:34:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:08 smithi169 bash[25083]: cluster 2023-12-16T07:34:07.292558+0000 mon.a (mon.0) 473 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2023-12-16T07:34:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:08 smithi169 bash[25083]: audit 2023-12-16T07:34:07.293431+0000 mon.a (mon.0) 474 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:34:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:08 smithi169 bash[25083]: audit 2023-12-16T07:34:07.305083+0000 mon.a (mon.0) 475 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:34:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:08 smithi169 bash[25083]: cluster 2023-12-16T07:34:07.391176+0000 mgr.y (mgr.14152) 194 : cluster [DBG] pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-12-16T07:34:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:08 smithi169 bash[25083]: cluster 2023-12-16T07:34:08.296105+0000 mon.a (mon.0) 476 : cluster [INF] osd.5 [v2:172.21.15.169:6808/3877143549,v1:172.21.15.169:6809/3877143549] boot 2023-12-16T07:34:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:08 smithi169 bash[25083]: cluster 2023-12-16T07:34:08.296252+0000 mon.a (mon.0) 477 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2023-12-16T07:34:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:08 smithi169 bash[25083]: audit 2023-12-16T07:34:08.296605+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:34:09.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:09 smithi007 bash[25009]: cluster 2023-12-16T07:34:06.920188+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:34:09.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:09 smithi007 bash[25009]: cluster 2023-12-16T07:34:06.920382+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:34:09.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:09 smithi007 bash[30436]: cluster 2023-12-16T07:34:06.920188+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:34:09.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:09 smithi007 bash[30436]: cluster 2023-12-16T07:34:06.920382+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:34:09.624 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-12-16T07:34:09.625 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-12-16T07:34:09.625 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2023-12-16T07:34:09.625 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2023-12-16T07:34:09.625 INFO:teuthology.orchestra.run.smithi169.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0146991 s, 713 MB/s 2023-12-16T07:34:09.625 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2023-12-16T07:34:09.667 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:09 smithi169 bash[25083]: cluster 2023-12-16T07:34:06.920188+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:34:09.667 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:09 smithi169 bash[25083]: cluster 2023-12-16T07:34:06.920382+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:34:10.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:10 smithi007 bash[25009]: cluster 2023-12-16T07:34:09.307493+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2023-12-16T07:34:10.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:10 smithi007 bash[25009]: cluster 2023-12-16T07:34:09.391905+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:10.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:10 smithi007 bash[30436]: cluster 2023-12-16T07:34:09.307493+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2023-12-16T07:34:10.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:10 smithi007 bash[30436]: cluster 2023-12-16T07:34:09.391905+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:10.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:10 smithi169 bash[25083]: cluster 2023-12-16T07:34:09.307493+0000 mon.a (mon.0) 479 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2023-12-16T07:34:10.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:10 smithi169 bash[25083]: cluster 2023-12-16T07:34:09.391905+0000 mgr.y (mgr.14152) 195 : cluster [DBG] pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:10.840 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_2 2023-12-16T07:34:11.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:11 smithi007 bash[25009]: cluster 2023-12-16T07:34:10.309696+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2023-12-16T07:34:11.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:11 smithi007 bash[25009]: cluster 2023-12-16T07:34:10.310565+0000 mon.a (mon.0) 481 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-12-16T07:34:11.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:11 smithi007 bash[30436]: cluster 2023-12-16T07:34:10.309696+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2023-12-16T07:34:11.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:11 smithi007 bash[30436]: cluster 2023-12-16T07:34:10.310565+0000 mon.a (mon.0) 481 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-12-16T07:34:11.573 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:11 smithi169 bash[25083]: cluster 2023-12-16T07:34:10.309696+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2023-12-16T07:34:11.573 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:11 smithi169 bash[25083]: cluster 2023-12-16T07:34:10.310565+0000 mon.a (mon.0) 481 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-12-16T07:34:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:12 smithi169 bash[25083]: cluster 2023-12-16T07:34:11.327651+0000 mon.a (mon.0) 482 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2023-12-16T07:34:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:12 smithi169 bash[25083]: cluster 2023-12-16T07:34:11.392612+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:12.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:12 smithi007 bash[25009]: cluster 2023-12-16T07:34:11.327651+0000 mon.a (mon.0) 482 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2023-12-16T07:34:12.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:12 smithi007 bash[25009]: cluster 2023-12-16T07:34:11.392612+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:12.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:12 smithi007 bash[30436]: cluster 2023-12-16T07:34:11.327651+0000 mon.a (mon.0) 482 : cluster [DBG] osdmap e41: 6 total, 6 up, 6 in 2023-12-16T07:34:12.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:12 smithi007 bash[30436]: cluster 2023-12-16T07:34:11.392612+0000 mgr.y (mgr.14152) 196 : cluster [DBG] pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:13.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:13 smithi169 bash[25083]: cluster 2023-12-16T07:34:12.336301+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2023-12-16T07:34:13.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:13 smithi007 bash[25009]: cluster 2023-12-16T07:34:12.336301+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2023-12-16T07:34:13.815 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:13 smithi007 bash[30436]: cluster 2023-12-16T07:34:12.336301+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e42: 6 total, 6 up, 6 in 2023-12-16T07:34:14.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:14 smithi169 bash[25083]: cluster 2023-12-16T07:34:13.393234+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2023-12-16T07:34:14.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:14 smithi169 bash[25083]: audit 2023-12-16T07:34:14.132312+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:14.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:14 smithi169 bash[25083]: audit 2023-12-16T07:34:14.134411+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:14.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:14 smithi169 bash[25083]: audit 2023-12-16T07:34:14.136144+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:14.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:14 smithi169 bash[25083]: audit 2023-12-16T07:34:14.142697+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:14.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:14 smithi169 bash[25083]: audit 2023-12-16T07:34:14.151295+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:14.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:14 smithi007 bash[25009]: cluster 2023-12-16T07:34:13.393234+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2023-12-16T07:34:14.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:14 smithi007 bash[25009]: audit 2023-12-16T07:34:14.132312+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:14.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:14 smithi007 bash[25009]: audit 2023-12-16T07:34:14.134411+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:14.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:14 smithi007 bash[25009]: audit 2023-12-16T07:34:14.136144+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:14.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:14 smithi007 bash[25009]: audit 2023-12-16T07:34:14.142697+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:14.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:14 smithi007 bash[25009]: audit 2023-12-16T07:34:14.151295+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:14.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:14 smithi007 bash[30436]: cluster 2023-12-16T07:34:13.393234+0000 mgr.y (mgr.14152) 197 : cluster [DBG] pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2023-12-16T07:34:14.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:14 smithi007 bash[30436]: audit 2023-12-16T07:34:14.132312+0000 mon.a (mon.0) 484 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:14.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:14 smithi007 bash[30436]: audit 2023-12-16T07:34:14.134411+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:14.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:14 smithi007 bash[30436]: audit 2023-12-16T07:34:14.136144+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:14.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:14 smithi007 bash[30436]: audit 2023-12-16T07:34:14.142697+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:14.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:14 smithi007 bash[30436]: audit 2023-12-16T07:34:14.151295+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:15.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:15 smithi169 bash[25083]: cephadm 2023-12-16T07:34:14.123178+0000 mgr.y (mgr.14152) 198 : cephadm [INF] Detected new or changed devices on smithi169 2023-12-16T07:34:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:15 smithi169 bash[25083]: cephadm 2023-12-16T07:34:14.137464+0000 mgr.y (mgr.14152) 199 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 8635M 2023-12-16T07:34:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:15 smithi169 bash[25083]: cluster 2023-12-16T07:34:14.335117+0000 mon.a (mon.0) 489 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-12-16T07:34:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:15 smithi169 bash[25083]: cluster 2023-12-16T07:34:14.335201+0000 mon.a (mon.0) 490 : cluster [INF] Cluster is now healthy 2023-12-16T07:34:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:15 smithi169 bash[25083]: audit 2023-12-16T07:34:14.552065+0000 mgr.y (mgr.14152) 200 : audit [DBG] from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:34:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:15 smithi169 bash[25083]: audit 2023-12-16T07:34:14.559110+0000 mon.a (mon.0) 491 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:34:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:15 smithi169 bash[25083]: audit 2023-12-16T07:34:14.564200+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:34:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:15 smithi169 bash[25083]: audit 2023-12-16T07:34:14.565589+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:15.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[25009]: cephadm 2023-12-16T07:34:14.123178+0000 mgr.y (mgr.14152) 198 : cephadm [INF] Detected new or changed devices on smithi169 2023-12-16T07:34:15.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[25009]: cephadm 2023-12-16T07:34:14.137464+0000 mgr.y (mgr.14152) 199 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 8635M 2023-12-16T07:34:15.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[25009]: cluster 2023-12-16T07:34:14.335117+0000 mon.a (mon.0) 489 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-12-16T07:34:15.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[25009]: cluster 2023-12-16T07:34:14.335201+0000 mon.a (mon.0) 490 : cluster [INF] Cluster is now healthy 2023-12-16T07:34:15.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[25009]: audit 2023-12-16T07:34:14.552065+0000 mgr.y (mgr.14152) 200 : audit [DBG] from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:34:15.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[25009]: audit 2023-12-16T07:34:14.559110+0000 mon.a (mon.0) 491 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:34:15.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[25009]: audit 2023-12-16T07:34:14.564200+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:34:15.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[25009]: audit 2023-12-16T07:34:14.565589+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:15.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[30436]: cephadm 2023-12-16T07:34:14.123178+0000 mgr.y (mgr.14152) 198 : cephadm [INF] Detected new or changed devices on smithi169 2023-12-16T07:34:15.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[30436]: cephadm 2023-12-16T07:34:14.137464+0000 mgr.y (mgr.14152) 199 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 8635M 2023-12-16T07:34:15.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[30436]: cluster 2023-12-16T07:34:14.335117+0000 mon.a (mon.0) 489 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-12-16T07:34:15.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[30436]: cluster 2023-12-16T07:34:14.335201+0000 mon.a (mon.0) 490 : cluster [INF] Cluster is now healthy 2023-12-16T07:34:15.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[30436]: audit 2023-12-16T07:34:14.552065+0000 mgr.y (mgr.14152) 200 : audit [DBG] from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:34:15.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[30436]: audit 2023-12-16T07:34:14.559110+0000 mon.a (mon.0) 491 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:34:15.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[30436]: audit 2023-12-16T07:34:14.564200+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:34:15.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:15 smithi007 bash[30436]: audit 2023-12-16T07:34:14.565589+0000 mon.a (mon.0) 493 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:16.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:16 smithi169 bash[25083]: cluster 2023-12-16T07:34:15.393852+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2023-12-16T07:34:16.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:16 smithi169 bash[25083]: audit 2023-12-16T07:34:15.395031+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:34:16.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:16 smithi169 bash[25083]: audit 2023-12-16T07:34:15.397925+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:34:16.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:16 smithi007 bash[25009]: cluster 2023-12-16T07:34:15.393852+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2023-12-16T07:34:16.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:16 smithi007 bash[25009]: audit 2023-12-16T07:34:15.395031+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:34:16.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:16 smithi007 bash[25009]: audit 2023-12-16T07:34:15.397925+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:34:16.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:16 smithi007 bash[30436]: cluster 2023-12-16T07:34:15.393852+0000 mgr.y (mgr.14152) 201 : cluster [DBG] pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2023-12-16T07:34:16.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:16 smithi007 bash[30436]: audit 2023-12-16T07:34:15.395031+0000 mon.a (mon.0) 494 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:34:16.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:16 smithi007 bash[30436]: audit 2023-12-16T07:34:15.397925+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:34:19.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:19 smithi007 bash[30436]: cluster 2023-12-16T07:34:17.394449+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-12-16T07:34:19.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:19 smithi007 bash[25009]: cluster 2023-12-16T07:34:17.394449+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-12-16T07:34:19.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:19 smithi169 bash[25083]: cluster 2023-12-16T07:34:17.394449+0000 mgr.y (mgr.14152) 202 : cluster [DBG] pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2023-12-16T07:34:21.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:21 smithi007 bash[25009]: cluster 2023-12-16T07:34:19.395076+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2023-12-16T07:34:21.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:21 smithi007 bash[25009]: audit 2023-12-16T07:34:20.540491+0000 mon.b (mon.2) 12 : audit [INF] from='client.? 172.21.15.169:0/397396501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75186dc4-13a9-4c53-9db6-07724e058c74"}]: dispatch 2023-12-16T07:34:21.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:21 smithi007 bash[25009]: audit 2023-12-16T07:34:20.541016+0000 mon.a (mon.0) 496 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75186dc4-13a9-4c53-9db6-07724e058c74"}]: dispatch 2023-12-16T07:34:21.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:21 smithi007 bash[25009]: audit 2023-12-16T07:34:20.551863+0000 mon.a (mon.0) 497 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75186dc4-13a9-4c53-9db6-07724e058c74"}]': finished 2023-12-16T07:34:21.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:21 smithi007 bash[25009]: cluster 2023-12-16T07:34:20.552154+0000 mon.a (mon.0) 498 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2023-12-16T07:34:21.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:21 smithi007 bash[25009]: audit 2023-12-16T07:34:20.552476+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:21.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:21 smithi007 bash[30436]: cluster 2023-12-16T07:34:19.395076+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2023-12-16T07:34:21.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:21 smithi007 bash[30436]: audit 2023-12-16T07:34:20.540491+0000 mon.b (mon.2) 12 : audit [INF] from='client.? 172.21.15.169:0/397396501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75186dc4-13a9-4c53-9db6-07724e058c74"}]: dispatch 2023-12-16T07:34:21.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:21 smithi007 bash[30436]: audit 2023-12-16T07:34:20.541016+0000 mon.a (mon.0) 496 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75186dc4-13a9-4c53-9db6-07724e058c74"}]: dispatch 2023-12-16T07:34:21.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:21 smithi007 bash[30436]: audit 2023-12-16T07:34:20.551863+0000 mon.a (mon.0) 497 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75186dc4-13a9-4c53-9db6-07724e058c74"}]': finished 2023-12-16T07:34:21.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:21 smithi007 bash[30436]: cluster 2023-12-16T07:34:20.552154+0000 mon.a (mon.0) 498 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2023-12-16T07:34:21.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:21 smithi007 bash[30436]: audit 2023-12-16T07:34:20.552476+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:21.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:21 smithi169 bash[25083]: cluster 2023-12-16T07:34:19.395076+0000 mgr.y (mgr.14152) 203 : cluster [DBG] pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2023-12-16T07:34:21.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:21 smithi169 bash[25083]: audit 2023-12-16T07:34:20.540491+0000 mon.b (mon.2) 12 : audit [INF] from='client.? 172.21.15.169:0/397396501' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75186dc4-13a9-4c53-9db6-07724e058c74"}]: dispatch 2023-12-16T07:34:21.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:21 smithi169 bash[25083]: audit 2023-12-16T07:34:20.541016+0000 mon.a (mon.0) 496 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75186dc4-13a9-4c53-9db6-07724e058c74"}]: dispatch 2023-12-16T07:34:21.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:21 smithi169 bash[25083]: audit 2023-12-16T07:34:20.551863+0000 mon.a (mon.0) 497 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75186dc4-13a9-4c53-9db6-07724e058c74"}]': finished 2023-12-16T07:34:21.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:21 smithi169 bash[25083]: cluster 2023-12-16T07:34:20.552154+0000 mon.a (mon.0) 498 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2023-12-16T07:34:21.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:21 smithi169 bash[25083]: audit 2023-12-16T07:34:20.552476+0000 mon.a (mon.0) 499 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:22.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:22 smithi169 bash[25083]: audit 2023-12-16T07:34:21.265885+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? 172.21.15.169:0/4248771641' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:34:22.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:22 smithi007 bash[25009]: audit 2023-12-16T07:34:21.265885+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? 172.21.15.169:0/4248771641' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:34:22.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:22 smithi007 bash[30436]: audit 2023-12-16T07:34:21.265885+0000 mon.b (mon.2) 13 : audit [DBG] from='client.? 172.21.15.169:0/4248771641' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:34:23.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:23 smithi169 bash[25083]: cluster 2023-12-16T07:34:21.395693+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-12-16T07:34:23.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:23 smithi007 bash[25009]: cluster 2023-12-16T07:34:21.395693+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-12-16T07:34:23.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:23 smithi007 bash[30436]: cluster 2023-12-16T07:34:21.395693+0000 mgr.y (mgr.14152) 204 : cluster [DBG] pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-12-16T07:34:25.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:25 smithi169 bash[25083]: cluster 2023-12-16T07:34:23.396315+0000 mgr.y (mgr.14152) 205 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:25.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:25 smithi007 bash[30436]: cluster 2023-12-16T07:34:23.396315+0000 mgr.y (mgr.14152) 205 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:25.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:25 smithi007 bash[25009]: cluster 2023-12-16T07:34:23.396315+0000 mgr.y (mgr.14152) 205 : cluster [DBG] pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:27.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:27 smithi169 bash[25083]: cluster 2023-12-16T07:34:25.396936+0000 mgr.y (mgr.14152) 206 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:27.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:27 smithi007 bash[30436]: cluster 2023-12-16T07:34:25.396936+0000 mgr.y (mgr.14152) 206 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:27.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:27 smithi007 bash[25009]: cluster 2023-12-16T07:34:25.396936+0000 mgr.y (mgr.14152) 206 : cluster [DBG] pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:29.370 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:29 smithi169 bash[25083]: cluster 2023-12-16T07:34:27.397555+0000 mgr.y (mgr.14152) 207 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:29.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:29 smithi007 bash[30436]: cluster 2023-12-16T07:34:27.397555+0000 mgr.y (mgr.14152) 207 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:29.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:29 smithi007 bash[25009]: cluster 2023-12-16T07:34:27.397555+0000 mgr.y (mgr.14152) 207 : cluster [DBG] pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:31.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:31 smithi169 bash[25083]: cluster 2023-12-16T07:34:29.398159+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:31.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:31 smithi169 bash[25083]: audit 2023-12-16T07:34:30.654038+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-12-16T07:34:31.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:31 smithi169 bash[25083]: audit 2023-12-16T07:34:30.655767+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:31.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:31 smithi007 bash[25009]: cluster 2023-12-16T07:34:29.398159+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:31.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:31 smithi007 bash[25009]: audit 2023-12-16T07:34:30.654038+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-12-16T07:34:31.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:31 smithi007 bash[25009]: audit 2023-12-16T07:34:30.655767+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:31.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:31 smithi007 bash[30436]: cluster 2023-12-16T07:34:29.398159+0000 mgr.y (mgr.14152) 208 : cluster [DBG] pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:31.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:31 smithi007 bash[30436]: audit 2023-12-16T07:34:30.654038+0000 mon.a (mon.0) 500 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-12-16T07:34:31.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:31 smithi007 bash[30436]: audit 2023-12-16T07:34:30.655767+0000 mon.a (mon.0) 501 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:32.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:32 smithi169 bash[25083]: cephadm 2023-12-16T07:34:30.657246+0000 mgr.y (mgr.14152) 209 : cephadm [INF] Deploying daemon osd.6 on smithi169 2023-12-16T07:34:32.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:32 smithi007 bash[25009]: cephadm 2023-12-16T07:34:30.657246+0000 mgr.y (mgr.14152) 209 : cephadm [INF] Deploying daemon osd.6 on smithi169 2023-12-16T07:34:32.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:32 smithi007 bash[30436]: cephadm 2023-12-16T07:34:30.657246+0000 mgr.y (mgr.14152) 209 : cephadm [INF] Deploying daemon osd.6 on smithi169 2023-12-16T07:34:33.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:33 smithi169 bash[25083]: cluster 2023-12-16T07:34:31.398724+0000 mgr.y (mgr.14152) 210 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:33.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:33 smithi007 bash[25009]: cluster 2023-12-16T07:34:31.398724+0000 mgr.y (mgr.14152) 210 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:33.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:33 smithi007 bash[30436]: cluster 2023-12-16T07:34:31.398724+0000 mgr.y (mgr.14152) 210 : cluster [DBG] pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:34.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:34 smithi169 bash[25083]: cluster 2023-12-16T07:34:33.399306+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:34.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:34 smithi007 bash[25009]: cluster 2023-12-16T07:34:33.399306+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:34.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:34 smithi007 bash[30436]: cluster 2023-12-16T07:34:33.399306+0000 mgr.y (mgr.14152) 211 : cluster [DBG] pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:36.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:35 smithi007 bash[25009]: audit 2023-12-16T07:34:34.746960+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:36.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:35 smithi007 bash[25009]: audit 2023-12-16T07:34:34.776136+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:34:36.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:35 smithi007 bash[25009]: audit 2023-12-16T07:34:34.778927+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:36.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:35 smithi007 bash[25009]: audit 2023-12-16T07:34:34.780621+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:34:36.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:35 smithi007 bash[30436]: audit 2023-12-16T07:34:34.746960+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:36.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:35 smithi007 bash[30436]: audit 2023-12-16T07:34:34.776136+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:34:36.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:35 smithi007 bash[30436]: audit 2023-12-16T07:34:34.778927+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:36.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:35 smithi007 bash[30436]: audit 2023-12-16T07:34:34.780621+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:34:36.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:35 smithi169 bash[25083]: audit 2023-12-16T07:34:34.746960+0000 mon.a (mon.0) 502 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:36.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:35 smithi169 bash[25083]: audit 2023-12-16T07:34:34.776136+0000 mon.a (mon.0) 503 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:34:36.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:35 smithi169 bash[25083]: audit 2023-12-16T07:34:34.778927+0000 mon.a (mon.0) 504 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:36.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:35 smithi169 bash[25083]: audit 2023-12-16T07:34:34.780621+0000 mon.a (mon.0) 505 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:34:36.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:36 smithi169 bash[25083]: cluster 2023-12-16T07:34:35.399885+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:37.065 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:36 smithi007 bash[25009]: cluster 2023-12-16T07:34:35.399885+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:37.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:36 smithi007 bash[30436]: cluster 2023-12-16T07:34:35.399885+0000 mgr.y (mgr.14152) 212 : cluster [DBG] pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:39.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:39 smithi169 bash[25083]: cluster 2023-12-16T07:34:37.400480+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:39.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:39 smithi169 bash[25083]: audit 2023-12-16T07:34:38.623092+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:39.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:39 smithi169 bash[25083]: audit 2023-12-16T07:34:38.907019+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:39.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:39 smithi007 bash[25009]: cluster 2023-12-16T07:34:37.400480+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:39.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:39 smithi007 bash[25009]: audit 2023-12-16T07:34:38.623092+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:39.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:39 smithi007 bash[25009]: audit 2023-12-16T07:34:38.907019+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:39.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:39 smithi007 bash[30436]: cluster 2023-12-16T07:34:37.400480+0000 mgr.y (mgr.14152) 213 : cluster [DBG] pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:39.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:39 smithi007 bash[30436]: audit 2023-12-16T07:34:38.623092+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:39.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:39 smithi007 bash[30436]: audit 2023-12-16T07:34:38.907019+0000 mon.a (mon.0) 507 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:41.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:41 smithi169 bash[25083]: cluster 2023-12-16T07:34:39.401106+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:41.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:41 smithi007 bash[25009]: cluster 2023-12-16T07:34:39.401106+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:41.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:41 smithi007 bash[30436]: cluster 2023-12-16T07:34:39.401106+0000 mgr.y (mgr.14152) 214 : cluster [DBG] pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:41.654 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 6 on host 'smithi169' 2023-12-16T07:34:42.356 DEBUG:teuthology.orchestra.run.smithi169:osd.6> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.6.service 2023-12-16T07:34:42.359 INFO:tasks.cephadm:Deploying osd.7 on smithi169 with /dev/vg_nvme/lv_1... 2023-12-16T07:34:42.359 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-12-16T07:34:42.571 INFO:journalctl@ceph.osd.6.smithi169.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:34:42.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:42 smithi169 bash[25083]: cluster 2023-12-16T07:34:41.401636+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:42.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:42 smithi169 bash[25083]: audit 2023-12-16T07:34:41.651817+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:42.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:42 smithi169 bash[25083]: audit 2023-12-16T07:34:41.683015+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:34:42.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:42 smithi169 bash[25083]: audit 2023-12-16T07:34:41.685923+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:42.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:42 smithi169 bash[25083]: audit 2023-12-16T07:34:41.687856+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:34:42.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:42 smithi169 bash[25083]: audit 2023-12-16T07:34:42.066210+0000 mon.b (mon.2) 14 : audit [INF] from='osd.6 [v2:172.21.15.169:6816/1373876707,v1:172.21.15.169:6817/1373876707]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-12-16T07:34:42.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:42 smithi169 bash[25083]: audit 2023-12-16T07:34:42.066408+0000 mon.a (mon.0) 512 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-12-16T07:34:43.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:42 smithi007 bash[25009]: cluster 2023-12-16T07:34:41.401636+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:43.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:42 smithi007 bash[25009]: audit 2023-12-16T07:34:41.651817+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:43.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:42 smithi007 bash[25009]: audit 2023-12-16T07:34:41.683015+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:34:43.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:42 smithi007 bash[25009]: audit 2023-12-16T07:34:41.685923+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:43.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:42 smithi007 bash[25009]: audit 2023-12-16T07:34:41.687856+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:34:43.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:42 smithi007 bash[25009]: audit 2023-12-16T07:34:42.066210+0000 mon.b (mon.2) 14 : audit [INF] from='osd.6 [v2:172.21.15.169:6816/1373876707,v1:172.21.15.169:6817/1373876707]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-12-16T07:34:43.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:42 smithi007 bash[25009]: audit 2023-12-16T07:34:42.066408+0000 mon.a (mon.0) 512 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-12-16T07:34:43.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:42 smithi007 bash[30436]: cluster 2023-12-16T07:34:41.401636+0000 mgr.y (mgr.14152) 215 : cluster [DBG] pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:43.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:42 smithi007 bash[30436]: audit 2023-12-16T07:34:41.651817+0000 mon.a (mon.0) 508 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:43.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:42 smithi007 bash[30436]: audit 2023-12-16T07:34:41.683015+0000 mon.a (mon.0) 509 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:34:43.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:42 smithi007 bash[30436]: audit 2023-12-16T07:34:41.685923+0000 mon.a (mon.0) 510 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:43.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:42 smithi007 bash[30436]: audit 2023-12-16T07:34:41.687856+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:34:43.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:42 smithi007 bash[30436]: audit 2023-12-16T07:34:42.066210+0000 mon.b (mon.2) 14 : audit [INF] from='osd.6 [v2:172.21.15.169:6816/1373876707,v1:172.21.15.169:6817/1373876707]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-12-16T07:34:43.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:42 smithi007 bash[30436]: audit 2023-12-16T07:34:42.066408+0000 mon.a (mon.0) 512 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-12-16T07:34:44.182 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:34:44 smithi169 bash[34375]: debug 2023-12-16T07:34:44.076+0000 7f83453a7700 -1 osd.6 0 waiting for initial osdmap 2023-12-16T07:34:44.183 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:34:44 smithi169 bash[34375]: debug 2023-12-16T07:34:44.092+0000 7f834053f700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:34:44.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:44 smithi169 bash[25083]: audit 2023-12-16T07:34:43.068210+0000 mon.a (mon.0) 513 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-12-16T07:34:44.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:44 smithi169 bash[25083]: cluster 2023-12-16T07:34:43.068455+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2023-12-16T07:34:44.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:44 smithi169 bash[25083]: audit 2023-12-16T07:34:43.068835+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:44.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:44 smithi169 bash[25083]: audit 2023-12-16T07:34:43.070256+0000 mon.b (mon.2) 15 : audit [INF] from='osd.6 [v2:172.21.15.169:6816/1373876707,v1:172.21.15.169:6817/1373876707]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:34:44.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:44 smithi169 bash[25083]: audit 2023-12-16T07:34:43.070510+0000 mon.a (mon.0) 516 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:34:44.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:44 smithi007 bash[25009]: audit 2023-12-16T07:34:43.068210+0000 mon.a (mon.0) 513 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-12-16T07:34:44.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:44 smithi007 bash[25009]: cluster 2023-12-16T07:34:43.068455+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2023-12-16T07:34:44.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:44 smithi007 bash[25009]: audit 2023-12-16T07:34:43.068835+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:44.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:44 smithi007 bash[25009]: audit 2023-12-16T07:34:43.070256+0000 mon.b (mon.2) 15 : audit [INF] from='osd.6 [v2:172.21.15.169:6816/1373876707,v1:172.21.15.169:6817/1373876707]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:34:44.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:44 smithi007 bash[25009]: audit 2023-12-16T07:34:43.070510+0000 mon.a (mon.0) 516 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:34:44.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:44 smithi007 bash[30436]: audit 2023-12-16T07:34:43.068210+0000 mon.a (mon.0) 513 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-12-16T07:34:44.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:44 smithi007 bash[30436]: cluster 2023-12-16T07:34:43.068455+0000 mon.a (mon.0) 514 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2023-12-16T07:34:44.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:44 smithi007 bash[30436]: audit 2023-12-16T07:34:43.068835+0000 mon.a (mon.0) 515 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:44.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:44 smithi007 bash[30436]: audit 2023-12-16T07:34:43.070256+0000 mon.b (mon.2) 15 : audit [INF] from='osd.6 [v2:172.21.15.169:6816/1373876707,v1:172.21.15.169:6817/1373876707]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:34:44.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:44 smithi007 bash[30436]: audit 2023-12-16T07:34:43.070510+0000 mon.a (mon.0) 516 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:34:45.218 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:45 smithi169 bash[25083]: cluster 2023-12-16T07:34:43.104948+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:34:45.218 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:45 smithi169 bash[25083]: cluster 2023-12-16T07:34:43.105177+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:34:45.218 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:45 smithi169 bash[25083]: cluster 2023-12-16T07:34:43.402321+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:45.219 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:45 smithi169 bash[25083]: audit 2023-12-16T07:34:44.071484+0000 mon.a (mon.0) 517 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T07:34:45.219 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:45 smithi169 bash[25083]: cluster 2023-12-16T07:34:44.071672+0000 mon.a (mon.0) 518 : cluster [DBG] osdmap e45: 7 total, 6 up, 7 in 2023-12-16T07:34:45.219 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:45 smithi169 bash[25083]: audit 2023-12-16T07:34:44.072328+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:45.219 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:45 smithi169 bash[25083]: audit 2023-12-16T07:34:44.080237+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:45.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:45 smithi007 bash[30436]: cluster 2023-12-16T07:34:43.104948+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:34:45.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:45 smithi007 bash[30436]: cluster 2023-12-16T07:34:43.105177+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:34:45.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:45 smithi007 bash[30436]: cluster 2023-12-16T07:34:43.402321+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:45.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:45 smithi007 bash[30436]: audit 2023-12-16T07:34:44.071484+0000 mon.a (mon.0) 517 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T07:34:45.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:45 smithi007 bash[30436]: cluster 2023-12-16T07:34:44.071672+0000 mon.a (mon.0) 518 : cluster [DBG] osdmap e45: 7 total, 6 up, 7 in 2023-12-16T07:34:45.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:45 smithi007 bash[30436]: audit 2023-12-16T07:34:44.072328+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:45.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:45 smithi007 bash[30436]: audit 2023-12-16T07:34:44.080237+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:45.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:45 smithi007 bash[25009]: cluster 2023-12-16T07:34:43.104948+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:34:45.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:45 smithi007 bash[25009]: cluster 2023-12-16T07:34:43.105177+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:34:45.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:45 smithi007 bash[25009]: cluster 2023-12-16T07:34:43.402321+0000 mgr.y (mgr.14152) 216 : cluster [DBG] pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-12-16T07:34:45.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:45 smithi007 bash[25009]: audit 2023-12-16T07:34:44.071484+0000 mon.a (mon.0) 517 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T07:34:45.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:45 smithi007 bash[25009]: cluster 2023-12-16T07:34:44.071672+0000 mon.a (mon.0) 518 : cluster [DBG] osdmap e45: 7 total, 6 up, 7 in 2023-12-16T07:34:45.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:45 smithi007 bash[25009]: audit 2023-12-16T07:34:44.072328+0000 mon.a (mon.0) 519 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:45.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:45 smithi007 bash[25009]: audit 2023-12-16T07:34:44.080237+0000 mon.a (mon.0) 520 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:46.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:46 smithi169 bash[25083]: cluster 2023-12-16T07:34:45.073207+0000 mon.a (mon.0) 521 : cluster [INF] osd.6 [v2:172.21.15.169:6816/1373876707,v1:172.21.15.169:6817/1373876707] boot 2023-12-16T07:34:46.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:46 smithi169 bash[25083]: cluster 2023-12-16T07:34:45.073347+0000 mon.a (mon.0) 522 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2023-12-16T07:34:46.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:46 smithi169 bash[25083]: audit 2023-12-16T07:34:45.073804+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:46.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:46 smithi169 bash[25083]: cluster 2023-12-16T07:34:46.075941+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2023-12-16T07:34:46.466 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-12-16T07:34:46.466 INFO:teuthology.orchestra.run.smithi169.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-12-16T07:34:46.467 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10+0 records in 2023-12-16T07:34:46.467 INFO:teuthology.orchestra.run.smithi169.stderr:10+0 records out 2023-12-16T07:34:46.467 INFO:teuthology.orchestra.run.smithi169.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0141936 s, 739 MB/s 2023-12-16T07:34:46.467 INFO:teuthology.orchestra.run.smithi169.stderr:--> Zapping successful for: 2023-12-16T07:34:46.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:46 smithi007 bash[25009]: cluster 2023-12-16T07:34:45.073207+0000 mon.a (mon.0) 521 : cluster [INF] osd.6 [v2:172.21.15.169:6816/1373876707,v1:172.21.15.169:6817/1373876707] boot 2023-12-16T07:34:46.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:46 smithi007 bash[25009]: cluster 2023-12-16T07:34:45.073347+0000 mon.a (mon.0) 522 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2023-12-16T07:34:46.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:46 smithi007 bash[25009]: audit 2023-12-16T07:34:45.073804+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:46.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:46 smithi007 bash[25009]: cluster 2023-12-16T07:34:46.075941+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2023-12-16T07:34:46.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:46 smithi007 bash[30436]: cluster 2023-12-16T07:34:45.073207+0000 mon.a (mon.0) 521 : cluster [INF] osd.6 [v2:172.21.15.169:6816/1373876707,v1:172.21.15.169:6817/1373876707] boot 2023-12-16T07:34:46.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:46 smithi007 bash[30436]: cluster 2023-12-16T07:34:45.073347+0000 mon.a (mon.0) 522 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2023-12-16T07:34:46.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:46 smithi007 bash[30436]: audit 2023-12-16T07:34:45.073804+0000 mon.a (mon.0) 523 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:34:46.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:46 smithi007 bash[30436]: cluster 2023-12-16T07:34:46.075941+0000 mon.a (mon.0) 524 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2023-12-16T07:34:47.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:47 smithi169 bash[25083]: cluster 2023-12-16T07:34:45.403069+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:47.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:47 smithi007 bash[25009]: cluster 2023-12-16T07:34:45.403069+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:47.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:47 smithi007 bash[30436]: cluster 2023-12-16T07:34:45.403069+0000 mgr.y (mgr.14152) 217 : cluster [DBG] pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:47.620 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch daemon add osd smithi169:vg_nvme/lv_1 2023-12-16T07:34:48.405 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:48 smithi169 bash[25083]: cluster 2023-12-16T07:34:47.085546+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2023-12-16T07:34:48.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:48 smithi007 bash[25009]: cluster 2023-12-16T07:34:47.085546+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2023-12-16T07:34:48.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:48 smithi007 bash[30436]: cluster 2023-12-16T07:34:47.085546+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e48: 7 total, 7 up, 7 in 2023-12-16T07:34:49.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:49 smithi169 bash[25083]: cluster 2023-12-16T07:34:47.403725+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:49.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:49 smithi007 bash[25009]: cluster 2023-12-16T07:34:47.403725+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:49.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:49 smithi007 bash[30436]: cluster 2023-12-16T07:34:47.403725+0000 mgr.y (mgr.14152) 218 : cluster [DBG] pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:50.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:50 smithi169 bash[25083]: cluster 2023-12-16T07:34:49.111809+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e49: 7 total, 7 up, 7 in 2023-12-16T07:34:50.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:50 smithi169 bash[25083]: cluster 2023-12-16T07:34:49.404429+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v206: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 119 KiB/s, 0 objects/s recovering 2023-12-16T07:34:50.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:50 smithi007 bash[25009]: cluster 2023-12-16T07:34:49.111809+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e49: 7 total, 7 up, 7 in 2023-12-16T07:34:50.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:50 smithi007 bash[25009]: cluster 2023-12-16T07:34:49.404429+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v206: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 119 KiB/s, 0 objects/s recovering 2023-12-16T07:34:50.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:50 smithi007 bash[30436]: cluster 2023-12-16T07:34:49.111809+0000 mon.a (mon.0) 526 : cluster [DBG] osdmap e49: 7 total, 7 up, 7 in 2023-12-16T07:34:50.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:50 smithi007 bash[30436]: cluster 2023-12-16T07:34:49.404429+0000 mgr.y (mgr.14152) 219 : cluster [DBG] pgmap v206: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 119 KiB/s, 0 objects/s recovering 2023-12-16T07:34:51.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:51 smithi169 bash[25083]: cluster 2023-12-16T07:34:50.116762+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e50: 7 total, 7 up, 7 in 2023-12-16T07:34:51.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:51 smithi007 bash[25009]: cluster 2023-12-16T07:34:50.116762+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e50: 7 total, 7 up, 7 in 2023-12-16T07:34:51.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:51 smithi007 bash[30436]: cluster 2023-12-16T07:34:50.116762+0000 mon.a (mon.0) 527 : cluster [DBG] osdmap e50: 7 total, 7 up, 7 in 2023-12-16T07:34:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:52 smithi169 bash[25083]: cephadm 2023-12-16T07:34:51.116050+0000 mgr.y (mgr.14152) 220 : cephadm [INF] Detected new or changed devices on smithi169 2023-12-16T07:34:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:52 smithi169 bash[25083]: audit 2023-12-16T07:34:51.125857+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:52 smithi169 bash[25083]: audit 2023-12-16T07:34:51.133071+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:52 smithi169 bash[25083]: audit 2023-12-16T07:34:51.134585+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:52 smithi169 bash[25083]: audit 2023-12-16T07:34:51.135945+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:52 smithi169 bash[25083]: cephadm 2023-12-16T07:34:51.136991+0000 mgr.y (mgr.14152) 221 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 5757M 2023-12-16T07:34:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:52 smithi169 bash[25083]: audit 2023-12-16T07:34:51.142953+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:52 smithi169 bash[25083]: audit 2023-12-16T07:34:51.150814+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:52 smithi169 bash[25083]: cluster 2023-12-16T07:34:51.405137+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v208: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:52 smithi169 bash[25083]: audit 2023-12-16T07:34:51.536623+0000 mgr.y (mgr.14152) 223 : audit [DBG] from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:34:52.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:52 smithi169 bash[25083]: audit 2023-12-16T07:34:51.543538+0000 mon.a (mon.0) 534 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:34:52.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:52 smithi169 bash[25083]: audit 2023-12-16T07:34:51.549987+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:34:52.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:52 smithi169 bash[25083]: audit 2023-12-16T07:34:51.551546+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:52.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[25009]: cephadm 2023-12-16T07:34:51.116050+0000 mgr.y (mgr.14152) 220 : cephadm [INF] Detected new or changed devices on smithi169 2023-12-16T07:34:52.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[25009]: audit 2023-12-16T07:34:51.125857+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:52.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[25009]: audit 2023-12-16T07:34:51.133071+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:52.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[25009]: audit 2023-12-16T07:34:51.134585+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:52.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[25009]: audit 2023-12-16T07:34:51.135945+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:52.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[25009]: cephadm 2023-12-16T07:34:51.136991+0000 mgr.y (mgr.14152) 221 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 5757M 2023-12-16T07:34:52.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[25009]: audit 2023-12-16T07:34:51.142953+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:52.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[25009]: audit 2023-12-16T07:34:51.150814+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:52.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[25009]: cluster 2023-12-16T07:34:51.405137+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v208: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:52.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[25009]: audit 2023-12-16T07:34:51.536623+0000 mgr.y (mgr.14152) 223 : audit [DBG] from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:34:52.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[25009]: audit 2023-12-16T07:34:51.543538+0000 mon.a (mon.0) 534 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:34:52.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[25009]: audit 2023-12-16T07:34:51.549987+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:34:52.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[25009]: audit 2023-12-16T07:34:51.551546+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:52.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[30436]: cephadm 2023-12-16T07:34:51.116050+0000 mgr.y (mgr.14152) 220 : cephadm [INF] Detected new or changed devices on smithi169 2023-12-16T07:34:52.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[30436]: audit 2023-12-16T07:34:51.125857+0000 mon.a (mon.0) 528 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:52.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[30436]: audit 2023-12-16T07:34:51.133071+0000 mon.a (mon.0) 529 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:52.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[30436]: audit 2023-12-16T07:34:51.134585+0000 mon.a (mon.0) 530 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:52.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[30436]: audit 2023-12-16T07:34:51.135945+0000 mon.a (mon.0) 531 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:34:52.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[30436]: cephadm 2023-12-16T07:34:51.136991+0000 mgr.y (mgr.14152) 221 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 5757M 2023-12-16T07:34:52.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[30436]: audit 2023-12-16T07:34:51.142953+0000 mon.a (mon.0) 532 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:52.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[30436]: audit 2023-12-16T07:34:51.150814+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:34:52.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[30436]: cluster 2023-12-16T07:34:51.405137+0000 mgr.y (mgr.14152) 222 : cluster [DBG] pgmap v208: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:52.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[30436]: audit 2023-12-16T07:34:51.536623+0000 mgr.y (mgr.14152) 223 : audit [DBG] from='client.24302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi169:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:34:52.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[30436]: audit 2023-12-16T07:34:51.543538+0000 mon.a (mon.0) 534 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-12-16T07:34:52.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[30436]: audit 2023-12-16T07:34:51.549987+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-12-16T07:34:52.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:52 smithi007 bash[30436]: audit 2023-12-16T07:34:51.551546+0000 mon.a (mon.0) 536 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:34:54.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:54 smithi169 bash[25083]: cluster 2023-12-16T07:34:53.405873+0000 mgr.y (mgr.14152) 224 : cluster [DBG] pgmap v209: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:54.815 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:54 smithi007 bash[25009]: cluster 2023-12-16T07:34:53.405873+0000 mgr.y (mgr.14152) 224 : cluster [DBG] pgmap v209: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:54.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:54 smithi007 bash[30436]: cluster 2023-12-16T07:34:53.405873+0000 mgr.y (mgr.14152) 224 : cluster [DBG] pgmap v209: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:56.815 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:56 smithi169 bash[25083]: cluster 2023-12-16T07:34:55.406698+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-12-16T07:34:56.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:56 smithi007 bash[25009]: cluster 2023-12-16T07:34:55.406698+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-12-16T07:34:56.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:56 smithi007 bash[30436]: cluster 2023-12-16T07:34:55.406698+0000 mgr.y (mgr.14152) 225 : cluster [DBG] pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2023-12-16T07:34:58.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:58 smithi169 bash[25083]: audit 2023-12-16T07:34:57.662543+0000 mon.b (mon.2) 16 : audit [INF] from='client.? 172.21.15.169:0/703956497' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f760eeb-da8d-4fce-a8f4-cf42d952fcc2"}]: dispatch 2023-12-16T07:34:58.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:58 smithi169 bash[25083]: audit 2023-12-16T07:34:57.662892+0000 mon.a (mon.0) 537 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f760eeb-da8d-4fce-a8f4-cf42d952fcc2"}]: dispatch 2023-12-16T07:34:58.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:58 smithi169 bash[25083]: audit 2023-12-16T07:34:57.671973+0000 mon.a (mon.0) 538 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f760eeb-da8d-4fce-a8f4-cf42d952fcc2"}]': finished 2023-12-16T07:34:58.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:58 smithi169 bash[25083]: cluster 2023-12-16T07:34:57.672123+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2023-12-16T07:34:58.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:58 smithi169 bash[25083]: audit 2023-12-16T07:34:57.672383+0000 mon.a (mon.0) 540 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:34:58.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:58 smithi007 bash[25009]: audit 2023-12-16T07:34:57.662543+0000 mon.b (mon.2) 16 : audit [INF] from='client.? 172.21.15.169:0/703956497' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f760eeb-da8d-4fce-a8f4-cf42d952fcc2"}]: dispatch 2023-12-16T07:34:58.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:58 smithi007 bash[25009]: audit 2023-12-16T07:34:57.662892+0000 mon.a (mon.0) 537 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f760eeb-da8d-4fce-a8f4-cf42d952fcc2"}]: dispatch 2023-12-16T07:34:58.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:58 smithi007 bash[25009]: audit 2023-12-16T07:34:57.671973+0000 mon.a (mon.0) 538 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f760eeb-da8d-4fce-a8f4-cf42d952fcc2"}]': finished 2023-12-16T07:34:58.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:58 smithi007 bash[25009]: cluster 2023-12-16T07:34:57.672123+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2023-12-16T07:34:58.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:58 smithi007 bash[25009]: audit 2023-12-16T07:34:57.672383+0000 mon.a (mon.0) 540 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:34:58.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:58 smithi007 bash[30436]: audit 2023-12-16T07:34:57.662543+0000 mon.b (mon.2) 16 : audit [INF] from='client.? 172.21.15.169:0/703956497' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f760eeb-da8d-4fce-a8f4-cf42d952fcc2"}]: dispatch 2023-12-16T07:34:58.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:58 smithi007 bash[30436]: audit 2023-12-16T07:34:57.662892+0000 mon.a (mon.0) 537 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f760eeb-da8d-4fce-a8f4-cf42d952fcc2"}]: dispatch 2023-12-16T07:34:58.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:58 smithi007 bash[30436]: audit 2023-12-16T07:34:57.671973+0000 mon.a (mon.0) 538 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f760eeb-da8d-4fce-a8f4-cf42d952fcc2"}]': finished 2023-12-16T07:34:58.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:58 smithi007 bash[30436]: cluster 2023-12-16T07:34:57.672123+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2023-12-16T07:34:58.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:58 smithi007 bash[30436]: audit 2023-12-16T07:34:57.672383+0000 mon.a (mon.0) 540 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:34:59.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:59 smithi169 bash[25083]: cluster 2023-12-16T07:34:57.407278+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:59.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:34:59 smithi169 bash[25083]: audit 2023-12-16T07:34:58.407916+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? 172.21.15.169:0/1732929108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:34:59.565 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:59 smithi007 bash[30436]: cluster 2023-12-16T07:34:57.407278+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:59.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:34:59 smithi007 bash[30436]: audit 2023-12-16T07:34:58.407916+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? 172.21.15.169:0/1732929108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:34:59.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:59 smithi007 bash[25009]: cluster 2023-12-16T07:34:57.407278+0000 mgr.y (mgr.14152) 226 : cluster [DBG] pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:34:59.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:34:59 smithi007 bash[25009]: audit 2023-12-16T07:34:58.407916+0000 mon.b (mon.2) 17 : audit [DBG] from='client.? 172.21.15.169:0/1732929108' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-12-16T07:35:01.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:01 smithi169 bash[25083]: cluster 2023-12-16T07:34:59.407899+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-12-16T07:35:01.387 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:01 smithi007 bash[30436]: cluster 2023-12-16T07:34:59.407899+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-12-16T07:35:01.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:01 smithi007 bash[25009]: cluster 2023-12-16T07:34:59.407899+0000 mgr.y (mgr.14152) 227 : cluster [DBG] pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2023-12-16T07:35:03.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:03 smithi169 bash[25083]: cluster 2023-12-16T07:35:01.408528+0000 mgr.y (mgr.14152) 228 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-12-16T07:35:03.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:03 smithi007 bash[25009]: cluster 2023-12-16T07:35:01.408528+0000 mgr.y (mgr.14152) 228 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-12-16T07:35:03.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:03 smithi007 bash[30436]: cluster 2023-12-16T07:35:01.408528+0000 mgr.y (mgr.14152) 228 : cluster [DBG] pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-12-16T07:35:05.320 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:05 smithi169 bash[25083]: cluster 2023-12-16T07:35:03.409277+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-12-16T07:35:05.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:05 smithi007 bash[25009]: cluster 2023-12-16T07:35:03.409277+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-12-16T07:35:05.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:05 smithi007 bash[30436]: cluster 2023-12-16T07:35:03.409277+0000 mgr.y (mgr.14152) 229 : cluster [DBG] pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-12-16T07:35:07.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:07 smithi169 bash[25083]: cluster 2023-12-16T07:35:05.409957+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:07.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:07 smithi007 bash[25009]: cluster 2023-12-16T07:35:05.409957+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:07.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:07 smithi007 bash[30436]: cluster 2023-12-16T07:35:05.409957+0000 mgr.y (mgr.14152) 230 : cluster [DBG] pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:09.409 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:09 smithi169 bash[25083]: cluster 2023-12-16T07:35:07.410655+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:09.409 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:09 smithi169 bash[25083]: audit 2023-12-16T07:35:08.230431+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-12-16T07:35:09.409 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:09 smithi169 bash[25083]: audit 2023-12-16T07:35:08.232151+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:09.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:09 smithi007 bash[25009]: cluster 2023-12-16T07:35:07.410655+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:09.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:09 smithi007 bash[25009]: audit 2023-12-16T07:35:08.230431+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-12-16T07:35:09.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:09 smithi007 bash[25009]: audit 2023-12-16T07:35:08.232151+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:09.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:09 smithi007 bash[30436]: cluster 2023-12-16T07:35:07.410655+0000 mgr.y (mgr.14152) 231 : cluster [DBG] pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:09.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:09 smithi007 bash[30436]: audit 2023-12-16T07:35:08.230431+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-12-16T07:35:09.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:09 smithi007 bash[30436]: audit 2023-12-16T07:35:08.232151+0000 mon.a (mon.0) 542 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:10.471 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:10 smithi169 bash[25083]: cephadm 2023-12-16T07:35:08.233525+0000 mgr.y (mgr.14152) 232 : cephadm [INF] Deploying daemon osd.7 on smithi169 2023-12-16T07:35:10.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:10 smithi007 bash[25009]: cephadm 2023-12-16T07:35:08.233525+0000 mgr.y (mgr.14152) 232 : cephadm [INF] Deploying daemon osd.7 on smithi169 2023-12-16T07:35:10.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:10 smithi007 bash[30436]: cephadm 2023-12-16T07:35:08.233525+0000 mgr.y (mgr.14152) 232 : cephadm [INF] Deploying daemon osd.7 on smithi169 2023-12-16T07:35:11.306 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:11 smithi169 bash[25083]: cluster 2023-12-16T07:35:09.411261+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:11.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:11 smithi007 bash[25009]: cluster 2023-12-16T07:35:09.411261+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:11.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:11 smithi007 bash[30436]: cluster 2023-12-16T07:35:09.411261+0000 mgr.y (mgr.14152) 233 : cluster [DBG] pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:13.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:13 smithi007 bash[30436]: cluster 2023-12-16T07:35:11.411874+0000 mgr.y (mgr.14152) 234 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:13.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:13 smithi007 bash[30436]: audit 2023-12-16T07:35:12.488917+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:13.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:13 smithi007 bash[30436]: audit 2023-12-16T07:35:12.528482+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:13.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:13 smithi007 bash[30436]: audit 2023-12-16T07:35:12.531046+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:13.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:13 smithi007 bash[30436]: audit 2023-12-16T07:35:12.532960+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:35:13.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:13 smithi007 bash[25009]: cluster 2023-12-16T07:35:11.411874+0000 mgr.y (mgr.14152) 234 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:13.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:13 smithi007 bash[25009]: audit 2023-12-16T07:35:12.488917+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:13.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:13 smithi007 bash[25009]: audit 2023-12-16T07:35:12.528482+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:13.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:13 smithi007 bash[25009]: audit 2023-12-16T07:35:12.531046+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:13.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:13 smithi007 bash[25009]: audit 2023-12-16T07:35:12.532960+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:35:13.429 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:13 smithi169 bash[25083]: cluster 2023-12-16T07:35:11.411874+0000 mgr.y (mgr.14152) 234 : cluster [DBG] pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:13.430 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:13 smithi169 bash[25083]: audit 2023-12-16T07:35:12.488917+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:13.430 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:13 smithi169 bash[25083]: audit 2023-12-16T07:35:12.528482+0000 mon.a (mon.0) 544 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:13.430 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:13 smithi169 bash[25083]: audit 2023-12-16T07:35:12.531046+0000 mon.a (mon.0) 545 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:13.430 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:13 smithi169 bash[25083]: audit 2023-12-16T07:35:12.532960+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:35:15.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:15 smithi169 bash[25083]: cluster 2023-12-16T07:35:13.412475+0000 mgr.y (mgr.14152) 235 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:15.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:15 smithi007 bash[25009]: cluster 2023-12-16T07:35:13.412475+0000 mgr.y (mgr.14152) 235 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:15.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:15 smithi007 bash[30436]: cluster 2023-12-16T07:35:13.412475+0000 mgr.y (mgr.14152) 235 : cluster [DBG] pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:16.377 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:16 smithi169 bash[25083]: audit 2023-12-16T07:35:15.399110+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:35:16.377 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:16 smithi169 bash[25083]: audit 2023-12-16T07:35:15.401882+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:35:16.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:16 smithi007 bash[25009]: audit 2023-12-16T07:35:15.399110+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:35:16.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:16 smithi007 bash[25009]: audit 2023-12-16T07:35:15.401882+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:35:16.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:16 smithi007 bash[30436]: audit 2023-12-16T07:35:15.399110+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:35:16.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:16 smithi007 bash[30436]: audit 2023-12-16T07:35:15.401882+0000 mon.a (mon.0) 548 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:35:17.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:17 smithi169 bash[25083]: cluster 2023-12-16T07:35:15.413060+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:17.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:17 smithi169 bash[25083]: audit 2023-12-16T07:35:16.582119+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:17.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:17 smithi169 bash[25083]: audit 2023-12-16T07:35:16.589080+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:17.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:17 smithi007 bash[25009]: cluster 2023-12-16T07:35:15.413060+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:17.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:17 smithi007 bash[25009]: audit 2023-12-16T07:35:16.582119+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:17.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:17 smithi007 bash[25009]: audit 2023-12-16T07:35:16.589080+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:17.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:17 smithi007 bash[30436]: cluster 2023-12-16T07:35:15.413060+0000 mgr.y (mgr.14152) 236 : cluster [DBG] pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:17.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:17 smithi007 bash[30436]: audit 2023-12-16T07:35:16.582119+0000 mon.a (mon.0) 549 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:17.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:17 smithi007 bash[30436]: audit 2023-12-16T07:35:16.589080+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:19.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:19 smithi169 bash[25083]: cluster 2023-12-16T07:35:17.413690+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:19.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:19 smithi007 bash[25009]: cluster 2023-12-16T07:35:17.413690+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:19.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:19 smithi007 bash[30436]: cluster 2023-12-16T07:35:17.413690+0000 mgr.y (mgr.14152) 237 : cluster [DBG] pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:19.604 INFO:teuthology.orchestra.run.smithi169.stdout:Created osd(s) 7 on host 'smithi169' 2023-12-16T07:35:20.291 DEBUG:teuthology.orchestra.run.smithi169:osd.7> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.7.service 2023-12-16T07:35:20.293 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2023-12-16T07:35:20.294 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph osd stat -f json 2023-12-16T07:35:20.500 INFO:journalctl@ceph.osd.7.smithi169.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:35:20.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:20 smithi007 bash[25009]: cluster 2023-12-16T07:35:19.414331+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:20.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:20 smithi007 bash[25009]: audit 2023-12-16T07:35:19.601509+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:20.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:20 smithi007 bash[25009]: audit 2023-12-16T07:35:19.617183+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:20.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:20 smithi007 bash[25009]: audit 2023-12-16T07:35:19.619586+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:20.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:20 smithi007 bash[25009]: audit 2023-12-16T07:35:19.621208+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:35:20.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:20 smithi007 bash[25009]: audit 2023-12-16T07:35:20.007911+0000 mon.b (mon.2) 18 : audit [INF] from='osd.7 [v2:172.21.15.169:6824/1772368148,v1:172.21.15.169:6825/1772368148]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-12-16T07:35:20.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:20 smithi007 bash[25009]: audit 2023-12-16T07:35:20.008115+0000 mon.a (mon.0) 555 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-12-16T07:35:20.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:20 smithi007 bash[30436]: cluster 2023-12-16T07:35:19.414331+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:20.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:20 smithi007 bash[30436]: audit 2023-12-16T07:35:19.601509+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:20.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:20 smithi007 bash[30436]: audit 2023-12-16T07:35:19.617183+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:20.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:20 smithi007 bash[30436]: audit 2023-12-16T07:35:19.619586+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:20.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:20 smithi007 bash[30436]: audit 2023-12-16T07:35:19.621208+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:35:20.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:20 smithi007 bash[30436]: audit 2023-12-16T07:35:20.007911+0000 mon.b (mon.2) 18 : audit [INF] from='osd.7 [v2:172.21.15.169:6824/1772368148,v1:172.21.15.169:6825/1772368148]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-12-16T07:35:20.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:20 smithi007 bash[30436]: audit 2023-12-16T07:35:20.008115+0000 mon.a (mon.0) 555 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-12-16T07:35:20.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:20 smithi169 bash[25083]: cluster 2023-12-16T07:35:19.414331+0000 mgr.y (mgr.14152) 238 : cluster [DBG] pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:20.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:20 smithi169 bash[25083]: audit 2023-12-16T07:35:19.601509+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:20.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:20 smithi169 bash[25083]: audit 2023-12-16T07:35:19.617183+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:20.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:20 smithi169 bash[25083]: audit 2023-12-16T07:35:19.619586+0000 mon.a (mon.0) 553 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:20.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:20 smithi169 bash[25083]: audit 2023-12-16T07:35:19.621208+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:35:20.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:20 smithi169 bash[25083]: audit 2023-12-16T07:35:20.007911+0000 mon.b (mon.2) 18 : audit [INF] from='osd.7 [v2:172.21.15.169:6824/1772368148,v1:172.21.15.169:6825/1772368148]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-12-16T07:35:20.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:20 smithi169 bash[25083]: audit 2023-12-16T07:35:20.008115+0000 mon.a (mon.0) 555 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-12-16T07:35:21.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:35:21 smithi169 bash[37512]: debug 2023-12-16T07:35:21.623+0000 7f74a38f9700 -1 osd.7 0 waiting for initial osdmap 2023-12-16T07:35:21.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:35:21 smithi169 bash[37512]: debug 2023-12-16T07:35:21.631+0000 7f749f292700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:35:21.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:21 smithi169 bash[25083]: audit 2023-12-16T07:35:20.611205+0000 mon.a (mon.0) 556 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-12-16T07:35:21.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:21 smithi169 bash[25083]: cluster 2023-12-16T07:35:20.611315+0000 mon.a (mon.0) 557 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2023-12-16T07:35:21.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:21 smithi169 bash[25083]: audit 2023-12-16T07:35:20.611495+0000 mon.a (mon.0) 558 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:21.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:21 smithi169 bash[25083]: audit 2023-12-16T07:35:20.614103+0000 mon.a (mon.0) 559 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:35:21.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:21 smithi169 bash[25083]: audit 2023-12-16T07:35:20.614138+0000 mon.b (mon.2) 19 : audit [INF] from='osd.7 [v2:172.21.15.169:6824/1772368148,v1:172.21.15.169:6825/1772368148]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:35:21.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:21 smithi007 bash[25009]: audit 2023-12-16T07:35:20.611205+0000 mon.a (mon.0) 556 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-12-16T07:35:21.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:21 smithi007 bash[25009]: cluster 2023-12-16T07:35:20.611315+0000 mon.a (mon.0) 557 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2023-12-16T07:35:21.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:21 smithi007 bash[25009]: audit 2023-12-16T07:35:20.611495+0000 mon.a (mon.0) 558 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:21.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:21 smithi007 bash[25009]: audit 2023-12-16T07:35:20.614103+0000 mon.a (mon.0) 559 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:35:21.971 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:21 smithi007 bash[25009]: audit 2023-12-16T07:35:20.614138+0000 mon.b (mon.2) 19 : audit [INF] from='osd.7 [v2:172.21.15.169:6824/1772368148,v1:172.21.15.169:6825/1772368148]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:35:21.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:21 smithi007 bash[30436]: audit 2023-12-16T07:35:20.611205+0000 mon.a (mon.0) 556 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-12-16T07:35:21.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:21 smithi007 bash[30436]: cluster 2023-12-16T07:35:20.611315+0000 mon.a (mon.0) 557 : cluster [DBG] osdmap e52: 8 total, 7 up, 8 in 2023-12-16T07:35:21.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:21 smithi007 bash[30436]: audit 2023-12-16T07:35:20.611495+0000 mon.a (mon.0) 558 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:21.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:21 smithi007 bash[30436]: audit 2023-12-16T07:35:20.614103+0000 mon.a (mon.0) 559 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:35:21.972 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:21 smithi007 bash[30436]: audit 2023-12-16T07:35:20.614138+0000 mon.b (mon.2) 19 : audit [INF] from='osd.7 [v2:172.21.15.169:6824/1772368148,v1:172.21.15.169:6825/1772368148]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:35:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:22 smithi169 bash[25083]: cluster 2023-12-16T07:35:21.414979+0000 mgr.y (mgr.14152) 239 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:22 smithi169 bash[25083]: audit 2023-12-16T07:35:21.616270+0000 mon.a (mon.0) 560 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T07:35:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:22 smithi169 bash[25083]: cluster 2023-12-16T07:35:21.616546+0000 mon.a (mon.0) 561 : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in 2023-12-16T07:35:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:22 smithi169 bash[25083]: audit 2023-12-16T07:35:21.617536+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:22 smithi169 bash[25083]: audit 2023-12-16T07:35:21.624900+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:22 smithi169 bash[25083]: cluster 2023-12-16T07:35:22.079613+0000 mon.a (mon.0) 564 : cluster [INF] osd.7 [v2:172.21.15.169:6824/1772368148,v1:172.21.15.169:6825/1772368148] boot 2023-12-16T07:35:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:22 smithi169 bash[25083]: cluster 2023-12-16T07:35:22.079668+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2023-12-16T07:35:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:22 smithi169 bash[25083]: audit 2023-12-16T07:35:22.080285+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:22.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[30436]: cluster 2023-12-16T07:35:21.414979+0000 mgr.y (mgr.14152) 239 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:22.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[30436]: audit 2023-12-16T07:35:21.616270+0000 mon.a (mon.0) 560 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T07:35:22.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[30436]: cluster 2023-12-16T07:35:21.616546+0000 mon.a (mon.0) 561 : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in 2023-12-16T07:35:22.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[30436]: audit 2023-12-16T07:35:21.617536+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:22.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[30436]: audit 2023-12-16T07:35:21.624900+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:22.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[30436]: cluster 2023-12-16T07:35:22.079613+0000 mon.a (mon.0) 564 : cluster [INF] osd.7 [v2:172.21.15.169:6824/1772368148,v1:172.21.15.169:6825/1772368148] boot 2023-12-16T07:35:22.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[30436]: cluster 2023-12-16T07:35:22.079668+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2023-12-16T07:35:22.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[30436]: audit 2023-12-16T07:35:22.080285+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:22.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[25009]: cluster 2023-12-16T07:35:21.414979+0000 mgr.y (mgr.14152) 239 : cluster [DBG] pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-12-16T07:35:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[25009]: audit 2023-12-16T07:35:21.616270+0000 mon.a (mon.0) 560 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]': finished 2023-12-16T07:35:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[25009]: cluster 2023-12-16T07:35:21.616546+0000 mon.a (mon.0) 561 : cluster [DBG] osdmap e53: 8 total, 7 up, 8 in 2023-12-16T07:35:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[25009]: audit 2023-12-16T07:35:21.617536+0000 mon.a (mon.0) 562 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[25009]: audit 2023-12-16T07:35:21.624900+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[25009]: cluster 2023-12-16T07:35:22.079613+0000 mon.a (mon.0) 564 : cluster [INF] osd.7 [v2:172.21.15.169:6824/1772368148,v1:172.21.15.169:6825/1772368148] boot 2023-12-16T07:35:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[25009]: cluster 2023-12-16T07:35:22.079668+0000 mon.a (mon.0) 565 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2023-12-16T07:35:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:22 smithi007 bash[25009]: audit 2023-12-16T07:35:22.080285+0000 mon.a (mon.0) 566 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:23.775 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-16T07:35:23.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:23 smithi169 bash[25083]: cluster 2023-12-16T07:35:20.968963+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:35:23.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:23 smithi169 bash[25083]: cluster 2023-12-16T07:35:20.969164+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:35:23.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:23 smithi169 bash[25083]: cluster 2023-12-16T07:35:23.089552+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2023-12-16T07:35:23.987 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:23 smithi007 bash[25009]: cluster 2023-12-16T07:35:20.968963+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:35:23.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:23 smithi007 bash[25009]: cluster 2023-12-16T07:35:20.969164+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:35:23.988 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:23 smithi007 bash[25009]: cluster 2023-12-16T07:35:23.089552+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2023-12-16T07:35:23.988 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:23 smithi007 bash[30436]: cluster 2023-12-16T07:35:20.968963+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2023-12-16T07:35:23.988 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:23 smithi007 bash[30436]: cluster 2023-12-16T07:35:20.969164+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2023-12-16T07:35:23.988 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:23 smithi007 bash[30436]: cluster 2023-12-16T07:35:23.089552+0000 mon.a (mon.0) 567 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2023-12-16T07:35:24.548 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1702712122,"num_in_osds":8,"osd_in_since":1702712097,"num_remapped_pgs":1} 2023-12-16T07:35:24.548 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-16T07:35:24.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:24 smithi007 bash[25009]: cluster 2023-12-16T07:35:23.415772+0000 mgr.y (mgr.14152) 240 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:24.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:24 smithi007 bash[25009]: audit 2023-12-16T07:35:23.774477+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.7:0/2265728622' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T07:35:24.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:24 smithi007 bash[25009]: cluster 2023-12-16T07:35:24.091978+0000 mon.a (mon.0) 568 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2023-12-16T07:35:24.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:24 smithi007 bash[30436]: cluster 2023-12-16T07:35:23.415772+0000 mgr.y (mgr.14152) 240 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:24.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:24 smithi007 bash[30436]: audit 2023-12-16T07:35:23.774477+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.7:0/2265728622' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T07:35:24.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:24 smithi007 bash[30436]: cluster 2023-12-16T07:35:24.091978+0000 mon.a (mon.0) 568 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2023-12-16T07:35:24.886 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:24 smithi169 bash[25083]: cluster 2023-12-16T07:35:23.415772+0000 mgr.y (mgr.14152) 240 : cluster [DBG] pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:24.886 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:24 smithi169 bash[25083]: audit 2023-12-16T07:35:23.774477+0000 mon.c (mon.1) 9 : audit [DBG] from='client.? 172.21.15.7:0/2265728622' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-12-16T07:35:24.886 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:24 smithi169 bash[25083]: cluster 2023-12-16T07:35:24.091978+0000 mon.a (mon.0) 568 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2023-12-16T07:35:26.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:26 smithi169 bash[25083]: cluster 2023-12-16T07:35:25.102178+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2023-12-16T07:35:26.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:26 smithi007 bash[25009]: cluster 2023-12-16T07:35:25.102178+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2023-12-16T07:35:26.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:26 smithi007 bash[30436]: cluster 2023-12-16T07:35:25.102178+0000 mon.a (mon.0) 569 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2023-12-16T07:35:27.279 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:35:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:27 smithi169 bash[25083]: cluster 2023-12-16T07:35:25.416431+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v232: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:27 smithi169 bash[25083]: cluster 2023-12-16T07:35:26.109969+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2023-12-16T07:35:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:27 smithi169 bash[25083]: cephadm 2023-12-16T07:35:26.778441+0000 mgr.y (mgr.14152) 242 : cephadm [INF] Detected new or changed devices on smithi169 2023-12-16T07:35:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:27 smithi169 bash[25083]: audit 2023-12-16T07:35:26.786040+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:27 smithi169 bash[25083]: audit 2023-12-16T07:35:26.787887+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:35:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:27 smithi169 bash[25083]: audit 2023-12-16T07:35:26.789227+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:35:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:27 smithi169 bash[25083]: audit 2023-12-16T07:35:26.790437+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:35:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:27 smithi169 bash[25083]: audit 2023-12-16T07:35:26.791580+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:35:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:27 smithi169 bash[25083]: cephadm 2023-12-16T07:35:26.792416+0000 mgr.y (mgr.14152) 243 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 4317M 2023-12-16T07:35:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:27 smithi169 bash[25083]: audit 2023-12-16T07:35:26.799343+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:27.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:27 smithi169 bash[25083]: audit 2023-12-16T07:35:26.806446+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:27.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[25009]: cluster 2023-12-16T07:35:25.416431+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v232: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:27.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[25009]: cluster 2023-12-16T07:35:26.109969+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2023-12-16T07:35:27.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[25009]: cephadm 2023-12-16T07:35:26.778441+0000 mgr.y (mgr.14152) 242 : cephadm [INF] Detected new or changed devices on smithi169 2023-12-16T07:35:27.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[25009]: audit 2023-12-16T07:35:26.786040+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:27.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[25009]: audit 2023-12-16T07:35:26.787887+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:35:27.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[25009]: audit 2023-12-16T07:35:26.789227+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:35:27.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[25009]: audit 2023-12-16T07:35:26.790437+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:35:27.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[25009]: audit 2023-12-16T07:35:26.791580+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:35:27.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[25009]: cephadm 2023-12-16T07:35:26.792416+0000 mgr.y (mgr.14152) 243 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 4317M 2023-12-16T07:35:27.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[25009]: audit 2023-12-16T07:35:26.799343+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:27.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[25009]: audit 2023-12-16T07:35:26.806446+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:27.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[30436]: cluster 2023-12-16T07:35:25.416431+0000 mgr.y (mgr.14152) 241 : cluster [DBG] pgmap v232: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:27.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[30436]: cluster 2023-12-16T07:35:26.109969+0000 mon.a (mon.0) 570 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2023-12-16T07:35:27.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[30436]: cephadm 2023-12-16T07:35:26.778441+0000 mgr.y (mgr.14152) 242 : cephadm [INF] Detected new or changed devices on smithi169 2023-12-16T07:35:27.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[30436]: audit 2023-12-16T07:35:26.786040+0000 mon.a (mon.0) 571 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:27.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[30436]: audit 2023-12-16T07:35:26.787887+0000 mon.a (mon.0) 572 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:35:27.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[30436]: audit 2023-12-16T07:35:26.789227+0000 mon.a (mon.0) 573 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:35:27.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[30436]: audit 2023-12-16T07:35:26.790437+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:35:27.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[30436]: audit 2023-12-16T07:35:26.791580+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:35:27.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[30436]: cephadm 2023-12-16T07:35:26.792416+0000 mgr.y (mgr.14152) 243 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 4317M 2023-12-16T07:35:27.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[30436]: audit 2023-12-16T07:35:26.799343+0000 mon.a (mon.0) 576 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:27.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:27 smithi007 bash[30436]: audit 2023-12-16T07:35:26.806446+0000 mon.a (mon.0) 577 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:28.420 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:28 smithi007 bash[25009]: cluster 2023-12-16T07:35:27.417059+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:28.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:28 smithi007 bash[30436]: cluster 2023-12-16T07:35:27.417059+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:28.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:28 smithi169 bash[25083]: cluster 2023-12-16T07:35:27.417059+0000 mgr.y (mgr.14152) 244 : cluster [DBG] pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:30.383 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-16T07:35:30.383 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":58,"fsid":"8b71d3dc-9be4-11ee-95a5-87774f69a715","created":"2023-12-16T07:28:10.358509+0000","modified":"2023-12-16T07:35:26.097316+0000","last_up_change":"2023-12-16T07:35:22.069655+0000","last_in_change":"2023-12-16T07:34:57.664166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T07:32:19.902470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d8aa3a96-fde6-48ee-9a0a-4ad9a0d4f427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3002763011},{"type":"v1","addr":"172.21.15.7:6803","nonce":3002763011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3002763011},{"type":"v1","addr":"172.21.15.7:6805","nonce":3002763011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3002763011},{"type":"v1","addr":"172.21.15.7:6809","nonce":3002763011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3002763011},{"type":"v1","addr":"172.21.15.7:6807","nonce":3002763011}]},"public_addr":"172.21.15.7:6803/3002763011","cluster_addr":"172.21.15.7:6805/3002763011","heartbeat_back_addr":"172.21.15.7:6809/3002763011","heartbeat_front_addr":"172.21.15.7:6807/3002763011","state":["exists","up"]},{"osd":1,"uuid":"b52565e1-b236-4f11-8356-c6e90f476298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":4053186404},{"type":"v1","addr":"172.21.15.7:6811","nonce":4053186404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":4053186404},{"type":"v1","addr":"172.21.15.7:6813","nonce":4053186404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":4053186404},{"type":"v1","addr":"172.21.15.7:6817","nonce":4053186404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":4053186404},{"type":"v1","addr":"172.21.15.7:6815","nonce":4053186404}]},"public_addr":"172.21.15.7:6811/4053186404","cluster_addr":"172.21.15.7:6813/4053186404","heartbeat_back_addr":"172.21.15.7:6817/4053186404","heartbeat_front_addr":"172.21.15.7:6815/4053186404","state":["exists","up"]},{"osd":2,"uuid":"5a43c2e0-1837-4880-b904-5ea6041d45a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":2173677438},{"type":"v1","addr":"172.21.15.7:6819","nonce":2173677438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":2173677438},{"type":"v1","addr":"172.21.15.7:6821","nonce":2173677438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":2173677438},{"type":"v1","addr":"172.21.15.7:6825","nonce":2173677438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":2173677438},{"type":"v1","addr":"172.21.15.7:6823","nonce":2173677438}]},"public_addr":"172.21.15.7:6819/2173677438","cluster_addr":"172.21.15.7:6821/2173677438","heartbeat_back_addr":"172.21.15.7:6825/2173677438","heartbeat_front_addr":"172.21.15.7:6823/2173677438","state":["exists","up"]},{"osd":3,"uuid":"4776e261-fa8a-4652-a32f-83710dda5dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1623735309},{"type":"v1","addr":"172.21.15.7:6827","nonce":1623735309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1623735309},{"type":"v1","addr":"172.21.15.7:6829","nonce":1623735309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1623735309},{"type":"v1","addr":"172.21.15.7:6833","nonce":1623735309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1623735309},{"type":"v1","addr":"172.21.15.7:6831","nonce":1623735309}]},"public_addr":"172.21.15.7:6827/1623735309","cluster_addr":"172.21.15.7:6829/1623735309","heartbeat_back_addr":"172.21.15.7:6833/1623735309","heartbeat_front_addr":"172.21.15.7:6831/1623735309","state":["exists","up"]},{"osd":4,"uuid":"d5631b50-8179-4ed7-8f03-4de60d19bb69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3747189467},{"type":"v1","addr":"172.21.15.169:6801","nonce":3747189467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3747189467},{"type":"v1","addr":"172.21.15.169:6803","nonce":3747189467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3747189467},{"type":"v1","addr":"172.21.15.169:6807","nonce":3747189467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3747189467},{"type":"v1","addr":"172.21.15.169:6805","nonce":3747189467}]},"public_addr":"172.21.15.169:6801/3747189467","cluster_addr":"172.21.15.169:6803/3747189467","heartbeat_back_addr":"172.21.15.169:6807/3747189467","heartbeat_front_addr":"172.21.15.169:6805/3747189467","state":["exists","up"]},{"osd":5,"uuid":"70f322c0-b8b5-439c-bf2a-8688a28ca919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":3877143549},{"type":"v1","addr":"172.21.15.169:6809","nonce":3877143549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":3877143549},{"type":"v1","addr":"172.21.15.169:6811","nonce":3877143549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3877143549},{"type":"v1","addr":"172.21.15.169:6815","nonce":3877143549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":3877143549},{"type":"v1","addr":"172.21.15.169:6813","nonce":3877143549}]},"public_addr":"172.21.15.169:6809/3877143549","cluster_addr":"172.21.15.169:6811/3877143549","heartbeat_back_addr":"172.21.15.169:6815/3877143549","heartbeat_front_addr":"172.21.15.169:6813/3877143549","state":["exists","up"]},{"osd":6,"uuid":"75186dc4-13a9-4c53-9db6-07724e058c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":1373876707},{"type":"v1","addr":"172.21.15.169:6817","nonce":1373876707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":1373876707},{"type":"v1","addr":"172.21.15.169:6819","nonce":1373876707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":1373876707},{"type":"v1","addr":"172.21.15.169:6823","nonce":1373876707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":1373876707},{"type":"v1","addr":"172.21.15.169:6821","nonce":1373876707}]},"public_addr":"172.21.15.169:6817/1373876707","cluster_addr":"172.21.15.169:6819/1373876707","heartbeat_back_addr":"172.21.15.169:6823/1373876707","heartbeat_front_addr":"172.21.15.169:6821/1373876707","state":["exists","up"]},{"osd":7,"uuid":"1f760eeb-da8d-4fce-a8f4-cf42d952fcc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1772368148},{"type":"v1","addr":"172.21.15.169:6825","nonce":1772368148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1772368148},{"type":"v1","addr":"172.21.15.169:6827","nonce":1772368148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1772368148},{"type":"v1","addr":"172.21.15.169:6831","nonce":1772368148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1772368148},{"type":"v1","addr":"172.21.15.169:6829","nonce":1772368148}]},"public_addr":"172.21.15.169:6825/1772368148","cluster_addr":"172.21.15.169:6827/1772368148","heartbeat_back_addr":"172.21.15.169:6831/1772368148","heartbeat_front_addr":"172.21.15.169:6829/1772368148","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:31:01.341467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:31:39.201921+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:32:17.444673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:32:56.146463+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:33:30.154703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:34:06.920388+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:34:43.105186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:35:20.969170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/2722441948":"2023-12-17T07:29:15.253163+0000","172.21.15.7:0/2431018109":"2023-12-17T07:29:15.253163+0000","172.21.15.7:6800/847724836":"2023-12-17T07:28:42.956756+0000","172.21.15.7:6801/2454016267":"2023-12-17T07:29:15.253163+0000","172.21.15.7:6801/847724836":"2023-12-17T07:28:42.956756+0000","172.21.15.7:0/1208557731":"2023-12-17T07:29:15.253163+0000","172.21.15.7:6800/2454016267":"2023-12-17T07:29:15.253163+0000","172.21.15.7:0/2162341476":"2023-12-17T07:28:42.956756+0000","172.21.15.7:0/3673321318":"2023-12-17T07:28:42.956756+0000","172.21.15.7:0/3086453482":"2023-12-17T07:28:42.956756+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T07:35:30.669 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:30 smithi007 bash[25009]: cluster 2023-12-16T07:35:29.417748+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2023-12-16T07:35:30.670 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:30 smithi007 bash[25009]: audit 2023-12-16T07:35:30.381601+0000 mon.a (mon.0) 578 : audit [DBG] from='client.? 172.21.15.7:0/1021055664' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T07:35:30.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:30 smithi007 bash[30436]: cluster 2023-12-16T07:35:29.417748+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2023-12-16T07:35:30.670 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:30 smithi007 bash[30436]: audit 2023-12-16T07:35:30.381601+0000 mon.a (mon.0) 578 : audit [DBG] from='client.? 172.21.15.7:0/1021055664' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T07:35:30.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:30 smithi169 bash[25083]: cluster 2023-12-16T07:35:29.417748+0000 mgr.y (mgr.14152) 245 : cluster [DBG] pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2023-12-16T07:35:30.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:30 smithi169 bash[25083]: audit 2023-12-16T07:35:30.381601+0000 mon.a (mon.0) 578 : audit [DBG] from='client.? 172.21.15.7:0/1021055664' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T07:35:31.277 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-16T07:32:19.902470+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-12-16T07:35:31.277 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph osd pool get .mgr pg_num 2023-12-16T07:35:33.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:33 smithi169 bash[25083]: cluster 2023-12-16T07:35:31.418342+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-12-16T07:35:33.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:33 smithi007 bash[25009]: cluster 2023-12-16T07:35:31.418342+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-12-16T07:35:33.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:33 smithi007 bash[30436]: cluster 2023-12-16T07:35:31.418342+0000 mgr.y (mgr.14152) 246 : cluster [DBG] pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-12-16T07:35:34.006 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:35:35.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:35 smithi007 bash[25009]: cluster 2023-12-16T07:35:33.419017+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-12-16T07:35:35.252 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:35 smithi007 bash[30436]: cluster 2023-12-16T07:35:33.419017+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-12-16T07:35:35.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:35 smithi169 bash[25083]: cluster 2023-12-16T07:35:33.419017+0000 mgr.y (mgr.14152) 247 : cluster [DBG] pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-12-16T07:35:36.969 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2023-12-16T07:35:37.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:37 smithi007 bash[25009]: cluster 2023-12-16T07:35:35.419660+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-12-16T07:35:37.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:37 smithi007 bash[25009]: audit 2023-12-16T07:35:36.968643+0000 mon.c (mon.1) 10 : audit [DBG] from='client.? 172.21.15.7:0/1153845425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-16T07:35:37.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:37 smithi007 bash[30436]: cluster 2023-12-16T07:35:35.419660+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-12-16T07:35:37.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:37 smithi007 bash[30436]: audit 2023-12-16T07:35:36.968643+0000 mon.c (mon.1) 10 : audit [DBG] from='client.? 172.21.15.7:0/1153845425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-16T07:35:37.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:37 smithi169 bash[25083]: cluster 2023-12-16T07:35:35.419660+0000 mgr.y (mgr.14152) 248 : cluster [DBG] pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-12-16T07:35:37.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:37 smithi169 bash[25083]: audit 2023-12-16T07:35:36.968643+0000 mon.c (mon.1) 10 : audit [DBG] from='client.? 172.21.15.7:0/1153845425' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-12-16T07:35:37.791 INFO:tasks.cephadm:Adding prometheus.a on smithi169 2023-12-16T07:35:37.792 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch apply prometheus '1;smithi169=a' 2023-12-16T07:35:39.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:39 smithi169 bash[25083]: cluster 2023-12-16T07:35:37.420324+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s, 0 objects/s recovering 2023-12-16T07:35:39.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:39 smithi007 bash[25009]: cluster 2023-12-16T07:35:37.420324+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s, 0 objects/s recovering 2023-12-16T07:35:39.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:39 smithi007 bash[30436]: cluster 2023-12-16T07:35:37.420324+0000 mgr.y (mgr.14152) 249 : cluster [DBG] pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s, 0 objects/s recovering 2023-12-16T07:35:40.679 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled prometheus update... 2023-12-16T07:35:41.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:41 smithi169 bash[25083]: cluster 2023-12-16T07:35:39.420955+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2023-12-16T07:35:41.614 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:41 smithi169 bash[25083]: audit 2023-12-16T07:35:40.677605+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:41.614 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:41 smithi169 bash[25083]: audit 2023-12-16T07:35:40.711612+0000 mon.a (mon.0) 580 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:41.614 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:41 smithi169 bash[25083]: audit 2023-12-16T07:35:40.714591+0000 mon.a (mon.0) 581 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:41.614 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:41 smithi169 bash[25083]: audit 2023-12-16T07:35:40.716765+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:35:41.614 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:41 smithi169 bash[25083]: audit 2023-12-16T07:35:41.002767+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:41.614 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:41 smithi169 bash[25083]: audit 2023-12-16T07:35:41.010569+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-16T07:35:41.615 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:41 smithi007 bash[25009]: cluster 2023-12-16T07:35:39.420955+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2023-12-16T07:35:41.615 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:41 smithi007 bash[25009]: audit 2023-12-16T07:35:40.677605+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:41.615 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:41 smithi007 bash[25009]: audit 2023-12-16T07:35:40.711612+0000 mon.a (mon.0) 580 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:41.615 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:41 smithi007 bash[25009]: audit 2023-12-16T07:35:40.714591+0000 mon.a (mon.0) 581 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:41.615 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:41 smithi007 bash[25009]: audit 2023-12-16T07:35:40.716765+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:35:41.615 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:41 smithi007 bash[25009]: audit 2023-12-16T07:35:41.002767+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:41.616 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:41 smithi007 bash[25009]: audit 2023-12-16T07:35:41.010569+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-16T07:35:41.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:41 smithi007 bash[30436]: cluster 2023-12-16T07:35:39.420955+0000 mgr.y (mgr.14152) 250 : cluster [DBG] pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2023-12-16T07:35:41.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:41 smithi007 bash[30436]: audit 2023-12-16T07:35:40.677605+0000 mon.a (mon.0) 579 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:41.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:41 smithi007 bash[30436]: audit 2023-12-16T07:35:40.711612+0000 mon.a (mon.0) 580 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:41.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:41 smithi007 bash[30436]: audit 2023-12-16T07:35:40.714591+0000 mon.a (mon.0) 581 : audit [DBG] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:41.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:41 smithi007 bash[30436]: audit 2023-12-16T07:35:40.716765+0000 mon.a (mon.0) 582 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:35:41.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:41 smithi007 bash[30436]: audit 2023-12-16T07:35:41.002767+0000 mon.a (mon.0) 583 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' 2023-12-16T07:35:41.616 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:41 smithi007 bash[30436]: audit 2023-12-16T07:35:41.010569+0000 mon.a (mon.0) 584 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-16T07:35:41.617 DEBUG:teuthology.orchestra.run.smithi169:prometheus.a> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@prometheus.a.service 2023-12-16T07:35:41.620 INFO:tasks.cephadm:Adding node-exporter.a on smithi007 2023-12-16T07:35:41.620 INFO:tasks.cephadm:Adding node-exporter.b on smithi169 2023-12-16T07:35:41.620 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch apply node-exporter '2;smithi007=a;smithi169=b' 2023-12-16T07:35:41.717 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:35:42.104 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:42 smithi169 bash[25855]: ignoring --setuser ceph since I am not root 2023-12-16T07:35:42.104 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:42 smithi169 bash[25855]: ignoring --setgroup ceph since I am not root 2023-12-16T07:35:42.104 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:42 smithi169 bash[25855]: debug 2023-12-16T07:35:42.054+0000 7f6b6b21d700 1 -- 172.21.15.169:0/897310264 <== mon.2 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d2a7254340 con 0x55d2a7fd0800 2023-12-16T07:35:42.316 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:42 smithi007 bash[25282]: ignoring --setuser ceph since I am not root 2023-12-16T07:35:42.316 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:42 smithi007 bash[25282]: ignoring --setgroup ceph since I am not root 2023-12-16T07:35:42.316 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:42 smithi007 bash[25282]: debug 2023-12-16T07:35:42.055+0000 7f7834b72700 1 -- 172.21.15.7:0/2898634609 <== mon.1 v2:172.21.15.7:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563d6ab04340 con 0x563d6b880800 2023-12-16T07:35:42.316 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:42 smithi007 bash[25282]: debug 2023-12-16T07:35:42.215+0000 7f783d5ce000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T07:35:42.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:42 smithi007 bash[30436]: audit 2023-12-16T07:35:40.666526+0000 mgr.y (mgr.14152) 251 : audit [DBG] from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi169=a", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:35:42.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:42 smithi007 bash[30436]: cephadm 2023-12-16T07:35:40.669961+0000 mgr.y (mgr.14152) 252 : cephadm [INF] Saving service prometheus spec with placement smithi169=a;count:1 2023-12-16T07:35:42.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:42 smithi007 bash[30436]: audit 2023-12-16T07:35:42.014917+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-16T07:35:42.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:42 smithi007 bash[30436]: cluster 2023-12-16T07:35:42.015016+0000 mon.a (mon.0) 586 : cluster [DBG] mgrmap e16: y(active, since 6m), standbys: x 2023-12-16T07:35:42.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:42 smithi007 bash[25009]: audit 2023-12-16T07:35:40.666526+0000 mgr.y (mgr.14152) 251 : audit [DBG] from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi169=a", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:35:42.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:42 smithi007 bash[25009]: cephadm 2023-12-16T07:35:40.669961+0000 mgr.y (mgr.14152) 252 : cephadm [INF] Saving service prometheus spec with placement smithi169=a;count:1 2023-12-16T07:35:42.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:42 smithi007 bash[25009]: audit 2023-12-16T07:35:42.014917+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-16T07:35:42.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:42 smithi007 bash[25009]: cluster 2023-12-16T07:35:42.015016+0000 mon.a (mon.0) 586 : cluster [DBG] mgrmap e16: y(active, since 6m), standbys: x 2023-12-16T07:35:42.432 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:42 smithi169 bash[25855]: debug 2023-12-16T07:35:42.214+0000 7f6b73c79000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T07:35:42.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:42 smithi169 bash[25083]: audit 2023-12-16T07:35:40.666526+0000 mgr.y (mgr.14152) 251 : audit [DBG] from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi169=a", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:35:42.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:42 smithi169 bash[25083]: cephadm 2023-12-16T07:35:40.669961+0000 mgr.y (mgr.14152) 252 : cephadm [INF] Saving service prometheus spec with placement smithi169=a;count:1 2023-12-16T07:35:42.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:42 smithi169 bash[25083]: audit 2023-12-16T07:35:42.014917+0000 mon.a (mon.0) 585 : audit [INF] from='mgr.14152 172.21.15.7:0/3683934259' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-16T07:35:42.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:42 smithi169 bash[25083]: cluster 2023-12-16T07:35:42.015016+0000 mon.a (mon.0) 586 : cluster [DBG] mgrmap e16: y(active, since 6m), standbys: x 2023-12-16T07:35:43.186 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:42 smithi169 bash[25855]: debug 2023-12-16T07:35:42.898+0000 7f6b73c79000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T07:35:43.186 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:43 smithi169 bash[25855]: debug 2023-12-16T07:35:43.022+0000 7f6b73c79000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T07:35:43.202 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:42 smithi007 bash[25282]: debug 2023-12-16T07:35:42.907+0000 7f783d5ce000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T07:35:43.203 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:43 smithi007 bash[25282]: debug 2023-12-16T07:35:43.031+0000 7f783d5ce000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T07:35:43.516 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:43 smithi169 bash[25855]: debug 2023-12-16T07:35:43.178+0000 7f6b73c79000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:35:43.544 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:43 smithi007 bash[25282]: debug 2023-12-16T07:35:43.195+0000 7f783d5ce000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:35:43.682 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:43 smithi169 bash[25855]: debug 2023-12-16T07:35:43.510+0000 7f6b73c79000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T07:35:43.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:43 smithi169 bash[25855]: debug 2023-12-16T07:35:43.602+0000 7f6b73c79000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T07:35:43.815 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:43 smithi007 bash[25282]: debug 2023-12-16T07:35:43.539+0000 7f783d5ce000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T07:35:43.816 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:43 smithi007 bash[25282]: debug 2023-12-16T07:35:43.635+0000 7f783d5ce000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T07:35:44.182 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:43 smithi169 bash[25855]: debug 2023-12-16T07:35:43.806+0000 7f6b73c79000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T07:35:44.316 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:43 smithi007 bash[25282]: debug 2023-12-16T07:35:43.843+0000 7f783d5ce000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T07:35:45.046 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:44 smithi169 bash[25855]: debug 2023-12-16T07:35:44.698+0000 7f6b73c79000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T07:35:45.046 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:44 smithi169 bash[25855]: debug 2023-12-16T07:35:44.814+0000 7f6b73c79000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T07:35:45.055 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:44 smithi007 bash[25282]: debug 2023-12-16T07:35:44.727+0000 7f783d5ce000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T07:35:45.055 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:44 smithi007 bash[25282]: debug 2023-12-16T07:35:44.839+0000 7f783d5ce000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T07:35:45.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:45 smithi007 bash[25282]: debug 2023-12-16T07:35:45.051+0000 7f783d5ce000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T07:35:45.398 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:45 smithi169 bash[25855]: debug 2023-12-16T07:35:45.042+0000 7f6b73c79000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T07:35:45.682 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:45 smithi169 bash[25855]: debug 2023-12-16T07:35:45.390+0000 7f6b73c79000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T07:35:45.757 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:45 smithi007 bash[25282]: debug 2023-12-16T07:35:45.411+0000 7f783d5ce000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T07:35:46.065 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:45 smithi007 bash[25282]: debug 2023-12-16T07:35:45.751+0000 7f783d5ce000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T07:35:46.091 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:45 smithi169 bash[25855]: debug 2023-12-16T07:35:45.722+0000 7f6b73c79000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T07:35:46.356 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:46 smithi169 bash[25855]: debug 2023-12-16T07:35:46.086+0000 7f6b73c79000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T07:35:46.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:46 smithi007 bash[25282]: debug 2023-12-16T07:35:46.139+0000 7f783d5ce000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T07:35:46.682 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:46 smithi169 bash[25855]: debug 2023-12-16T07:35:46.350+0000 7f6b73c79000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:35:46.815 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:46 smithi007 bash[25282]: debug 2023-12-16T07:35:46.419+0000 7f783d5ce000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:35:47.288 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:47 smithi169 bash[25855]: debug 2023-12-16T07:35:47.022+0000 7f6b73c79000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T07:35:47.288 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:47 smithi169 bash[25855]: debug 2023-12-16T07:35:47.130+0000 7f6b73c79000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T07:35:47.406 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:47 smithi007 bash[25282]: debug 2023-12-16T07:35:47.127+0000 7f783d5ce000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T07:35:47.406 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:47 smithi007 bash[25282]: debug 2023-12-16T07:35:47.235+0000 7f783d5ce000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T07:35:47.661 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:47 smithi007 bash[25282]: debug 2023-12-16T07:35:47.399+0000 7f783d5ce000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T07:35:47.661 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:47 smithi007 bash[25282]: debug 2023-12-16T07:35:47.511+0000 7f783d5ce000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T07:35:47.682 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:47 smithi169 bash[25855]: debug 2023-12-16T07:35:47.282+0000 7f6b73c79000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T07:35:47.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:47 smithi169 bash[25855]: debug 2023-12-16T07:35:47.386+0000 7f6b73c79000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T07:35:47.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:47 smithi169 bash[25855]: debug 2023-12-16T07:35:47.522+0000 7f6b73c79000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T07:35:47.959 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:47 smithi007 bash[25282]: debug 2023-12-16T07:35:47.655+0000 7f783d5ce000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T07:35:48.112 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:47 smithi169 bash[25855]: debug 2023-12-16T07:35:47.814+0000 7f6b73c79000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T07:35:48.112 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:47 smithi169 bash[25855]: debug 2023-12-16T07:35:47.930+0000 7f6b73c79000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T07:35:48.274 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:47 smithi007 bash[25282]: debug 2023-12-16T07:35:47.951+0000 7f783d5ce000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T07:35:48.275 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:48 smithi007 bash[25282]: debug 2023-12-16T07:35:48.075+0000 7f783d5ce000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T07:35:48.432 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:48 smithi169 bash[25855]: debug 2023-12-16T07:35:48.106+0000 7f6b73c79000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T07:35:48.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:48 smithi169 bash[25855]: debug 2023-12-16T07:35:48.222+0000 7f6b73c79000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T07:35:48.566 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:48 smithi007 bash[25282]: debug 2023-12-16T07:35:48.267+0000 7f783d5ce000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T07:35:48.566 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:48 smithi007 bash[25282]: debug 2023-12-16T07:35:48.395+0000 7f783d5ce000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T07:35:49.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:49 smithi169 bash[25083]: cluster 2023-12-16T07:35:49.088966+0000 mon.a (mon.0) 587 : cluster [DBG] Standby manager daemon x restarted 2023-12-16T07:35:49.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:49 smithi169 bash[25083]: cluster 2023-12-16T07:35:49.089271+0000 mon.a (mon.0) 588 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:35:49.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:49 smithi169 bash[25083]: audit 2023-12-16T07:35:49.094140+0000 mon.b (mon.2) 20 : audit [DBG] from='mgr.? 172.21.15.169:0/3971541379' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:35:49.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:49 smithi169 bash[25083]: audit 2023-12-16T07:35:49.095227+0000 mon.b (mon.2) 21 : audit [DBG] from='mgr.? 172.21.15.169:0/3971541379' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:35:49.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:49 smithi169 bash[25083]: audit 2023-12-16T07:35:49.100211+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.169:0/3971541379' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:35:49.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:49 smithi169 bash[25083]: audit 2023-12-16T07:35:49.100840+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.169:0/3971541379' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:35:49.434 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:49 smithi169 bash[25855]: debug 2023-12-16T07:35:49.058+0000 7f6b73c79000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T07:35:49.434 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:49 smithi169 bash[25855]: [16/Dec/2023:07:35:49] ENGINE Bus STARTING 2023-12-16T07:35:49.435 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:49 smithi169 bash[25855]: CherryPy Checker: 2023-12-16T07:35:49.435 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:49 smithi169 bash[25855]: The Application mounted at '' has an empty config. 2023-12-16T07:35:49.435 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:49 smithi169 bash[25855]: [16/Dec/2023:07:35:49] ENGINE Serving on http://:::9283 2023-12-16T07:35:49.435 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:35:49 smithi169 bash[25855]: [16/Dec/2023:07:35:49] ENGINE Bus STARTED 2023-12-16T07:35:49.566 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:49 smithi007 bash[25282]: debug 2023-12-16T07:35:49.287+0000 7f783d5ce000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T07:35:49.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:49 smithi007 bash[30436]: cluster 2023-12-16T07:35:49.088966+0000 mon.a (mon.0) 587 : cluster [DBG] Standby manager daemon x restarted 2023-12-16T07:35:49.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:49 smithi007 bash[30436]: cluster 2023-12-16T07:35:49.089271+0000 mon.a (mon.0) 588 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:35:49.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:49 smithi007 bash[30436]: audit 2023-12-16T07:35:49.094140+0000 mon.b (mon.2) 20 : audit [DBG] from='mgr.? 172.21.15.169:0/3971541379' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:35:49.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:49 smithi007 bash[30436]: audit 2023-12-16T07:35:49.095227+0000 mon.b (mon.2) 21 : audit [DBG] from='mgr.? 172.21.15.169:0/3971541379' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:35:49.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:49 smithi007 bash[30436]: audit 2023-12-16T07:35:49.100211+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.169:0/3971541379' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:35:49.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:49 smithi007 bash[30436]: audit 2023-12-16T07:35:49.100840+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.169:0/3971541379' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:35:49.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:49 smithi007 bash[25009]: cluster 2023-12-16T07:35:49.088966+0000 mon.a (mon.0) 587 : cluster [DBG] Standby manager daemon x restarted 2023-12-16T07:35:49.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:49 smithi007 bash[25009]: cluster 2023-12-16T07:35:49.089271+0000 mon.a (mon.0) 588 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:35:49.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:49 smithi007 bash[25009]: audit 2023-12-16T07:35:49.094140+0000 mon.b (mon.2) 20 : audit [DBG] from='mgr.? 172.21.15.169:0/3971541379' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:35:49.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:49 smithi007 bash[25009]: audit 2023-12-16T07:35:49.095227+0000 mon.b (mon.2) 21 : audit [DBG] from='mgr.? 172.21.15.169:0/3971541379' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:35:49.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:49 smithi007 bash[25009]: audit 2023-12-16T07:35:49.100211+0000 mon.b (mon.2) 22 : audit [DBG] from='mgr.? 172.21.15.169:0/3971541379' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:35:49.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:49 smithi007 bash[25009]: audit 2023-12-16T07:35:49.100840+0000 mon.b (mon.2) 23 : audit [DBG] from='mgr.? 172.21.15.169:0/3971541379' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:35:50.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:50 smithi169 bash[25083]: cluster 2023-12-16T07:35:49.151133+0000 mon.a (mon.0) 589 : cluster [DBG] mgrmap e17: y(active, since 6m), standbys: x 2023-12-16T07:35:50.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:50 smithi169 bash[25083]: cluster 2023-12-16T07:35:49.313726+0000 mon.a (mon.0) 590 : cluster [INF] Active manager daemon y restarted 2023-12-16T07:35:50.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:50 smithi169 bash[25083]: cluster 2023-12-16T07:35:49.315019+0000 mon.a (mon.0) 591 : cluster [INF] Activating manager daemon y 2023-12-16T07:35:50.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:50 smithi169 bash[25083]: cluster 2023-12-16T07:35:49.321314+0000 mon.a (mon.0) 592 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2023-12-16T07:35:50.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:50 smithi007 bash[25009]: cluster 2023-12-16T07:35:49.151133+0000 mon.a (mon.0) 589 : cluster [DBG] mgrmap e17: y(active, since 6m), standbys: x 2023-12-16T07:35:50.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:50 smithi007 bash[25009]: cluster 2023-12-16T07:35:49.313726+0000 mon.a (mon.0) 590 : cluster [INF] Active manager daemon y restarted 2023-12-16T07:35:50.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:50 smithi007 bash[25009]: cluster 2023-12-16T07:35:49.315019+0000 mon.a (mon.0) 591 : cluster [INF] Activating manager daemon y 2023-12-16T07:35:50.559 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:50 smithi007 bash[25009]: cluster 2023-12-16T07:35:49.321314+0000 mon.a (mon.0) 592 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2023-12-16T07:35:50.560 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:50 smithi007 bash[30436]: cluster 2023-12-16T07:35:49.151133+0000 mon.a (mon.0) 589 : cluster [DBG] mgrmap e17: y(active, since 6m), standbys: x 2023-12-16T07:35:50.560 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:50 smithi007 bash[30436]: cluster 2023-12-16T07:35:49.313726+0000 mon.a (mon.0) 590 : cluster [INF] Active manager daemon y restarted 2023-12-16T07:35:50.560 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:50 smithi007 bash[30436]: cluster 2023-12-16T07:35:49.315019+0000 mon.a (mon.0) 591 : cluster [INF] Activating manager daemon y 2023-12-16T07:35:50.560 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:50 smithi007 bash[30436]: cluster 2023-12-16T07:35:49.321314+0000 mon.a (mon.0) 592 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2023-12-16T07:35:50.560 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:50 smithi007 bash[25282]: [16/Dec/2023:07:35:50] ENGINE Bus STARTING 2023-12-16T07:35:50.815 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:50 smithi007 bash[25282]: CherryPy Checker: 2023-12-16T07:35:50.816 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:50 smithi007 bash[25282]: The Application mounted at '' has an empty config. 2023-12-16T07:35:50.816 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:50 smithi007 bash[25282]: [16/Dec/2023:07:35:50] ENGINE Serving on http://:::9283 2023-12-16T07:35:50.816 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:50 smithi007 bash[25282]: [16/Dec/2023:07:35:50] ENGINE Bus STARTED 2023-12-16T07:35:51.186 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: cluster 2023-12-16T07:35:50.171376+0000 mon.a (mon.0) 593 : cluster [DBG] mgrmap e18: y(active, starting, since 0.856598s), standbys: x 2023-12-16T07:35:51.196 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled node-exporter update... 2023-12-16T07:35:51.442 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: cluster 2023-12-16T07:35:50.171376+0000 mon.a (mon.0) 593 : cluster [DBG] mgrmap e18: y(active, starting, since 0.856598s), standbys: x 2023-12-16T07:35:51.442 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.176375+0000 mon.a (mon.0) 594 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:35:51.442 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.176575+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:35:51.443 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.176726+0000 mon.a (mon.0) 596 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:35:51.443 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.178813+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:35:51.443 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.179076+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:35:51.443 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.179453+0000 mon.a (mon.0) 599 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:35:51.443 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.179802+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:35:51.443 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.180137+0000 mon.a (mon.0) 601 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:35:51.443 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.180476+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:35:51.443 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.180866+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:35:51.443 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.181187+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:35:51.443 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.181419+0000 mon.a (mon.0) 605 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:35:51.443 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.181750+0000 mon.a (mon.0) 606 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:51.443 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.182009+0000 mon.a (mon.0) 607 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:35:51.443 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.182263+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:35:51.443 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.182808+0000 mon.a (mon.0) 609 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:35:51.444 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: cluster 2023-12-16T07:35:50.192565+0000 mon.a (mon.0) 610 : cluster [INF] Manager daemon y is now available 2023-12-16T07:35:51.444 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.226100+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:51.444 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.232467+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:51.444 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.235471+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:51.444 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.239203+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:35:51.444 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.264477+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:51.444 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.516128+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:35:51.444 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:51 smithi169 bash[25083]: audit 2023-12-16T07:35:50.528546+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:35:51.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: cluster 2023-12-16T07:35:50.171376+0000 mon.a (mon.0) 593 : cluster [DBG] mgrmap e18: y(active, starting, since 0.856598s), standbys: x 2023-12-16T07:35:51.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.176375+0000 mon.a (mon.0) 594 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:35:51.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.176575+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:35:51.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.176726+0000 mon.a (mon.0) 596 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:35:51.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.178813+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:35:51.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.179076+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:35:51.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.179453+0000 mon.a (mon.0) 599 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:35:51.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.179802+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:35:51.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.180137+0000 mon.a (mon.0) 601 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:35:51.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.180476+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:35:51.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.180866+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:35:51.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.181187+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:35:51.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.181419+0000 mon.a (mon.0) 605 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:35:51.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.181750+0000 mon.a (mon.0) 606 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:51.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.182009+0000 mon.a (mon.0) 607 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:35:51.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.182263+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:35:51.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.182808+0000 mon.a (mon.0) 609 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:35:51.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: cluster 2023-12-16T07:35:50.192565+0000 mon.a (mon.0) 610 : cluster [INF] Manager daemon y is now available 2023-12-16T07:35:51.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.226100+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:51.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.232467+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:51.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.235471+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:51.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.239203+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:35:51.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.264477+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:51.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.516128+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:35:51.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[30436]: audit 2023-12-16T07:35:50.528546+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:35:51.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.176375+0000 mon.a (mon.0) 594 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:35:51.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.176575+0000 mon.a (mon.0) 595 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:35:51.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.176726+0000 mon.a (mon.0) 596 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:35:51.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.178813+0000 mon.a (mon.0) 597 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:35:51.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.179076+0000 mon.a (mon.0) 598 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:35:51.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.179453+0000 mon.a (mon.0) 599 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:35:51.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.179802+0000 mon.a (mon.0) 600 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:35:51.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.180137+0000 mon.a (mon.0) 601 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:35:51.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.180476+0000 mon.a (mon.0) 602 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:35:51.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.180866+0000 mon.a (mon.0) 603 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:35:51.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.181187+0000 mon.a (mon.0) 604 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:35:51.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.181419+0000 mon.a (mon.0) 605 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:35:51.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.181750+0000 mon.a (mon.0) 606 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:35:51.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.182009+0000 mon.a (mon.0) 607 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:35:51.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.182263+0000 mon.a (mon.0) 608 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:35:51.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.182808+0000 mon.a (mon.0) 609 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:35:51.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: cluster 2023-12-16T07:35:50.192565+0000 mon.a (mon.0) 610 : cluster [INF] Manager daemon y is now available 2023-12-16T07:35:51.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.226100+0000 mon.a (mon.0) 611 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:51.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.232467+0000 mon.a (mon.0) 612 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:35:51.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.235471+0000 mon.a (mon.0) 613 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:35:51.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.239203+0000 mon.a (mon.0) 614 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:35:51.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.264477+0000 mon.a (mon.0) 615 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:51.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.516128+0000 mon.a (mon.0) 616 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:35:51.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:51 smithi007 bash[25009]: audit 2023-12-16T07:35:50.528546+0000 mon.a (mon.0) 617 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:35:52.018 DEBUG:teuthology.orchestra.run.smithi007:node-exporter.a> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@node-exporter.a.service 2023-12-16T07:35:52.021 DEBUG:teuthology.orchestra.run.smithi169:node-exporter.b> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@node-exporter.b.service 2023-12-16T07:35:52.023 INFO:tasks.cephadm:Adding alertmanager.a on smithi007 2023-12-16T07:35:52.023 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch apply alertmanager '1;smithi007=a' 2023-12-16T07:35:52.222 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:35:52.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:52 smithi007 bash[30436]: cluster 2023-12-16T07:35:51.182421+0000 mon.a (mon.0) 618 : cluster [DBG] mgrmap e19: y(active, since 1.86764s), standbys: x 2023-12-16T07:35:52.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:52 smithi007 bash[30436]: cephadm 2023-12-16T07:35:51.191999+0000 mgr.y (mgr.24367) 2 : cephadm [INF] Saving service node-exporter spec with placement smithi007=a;smithi169=b;count:2 2023-12-16T07:35:52.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:52 smithi007 bash[30436]: cluster 2023-12-16T07:35:51.192092+0000 mgr.y (mgr.24367) 3 : cluster [DBG] pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:52.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:52 smithi007 bash[30436]: audit 2023-12-16T07:35:51.195362+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:52.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:52 smithi007 bash[25009]: cluster 2023-12-16T07:35:51.182421+0000 mon.a (mon.0) 618 : cluster [DBG] mgrmap e19: y(active, since 1.86764s), standbys: x 2023-12-16T07:35:52.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:52 smithi007 bash[25009]: cephadm 2023-12-16T07:35:51.191999+0000 mgr.y (mgr.24367) 2 : cephadm [INF] Saving service node-exporter spec with placement smithi007=a;smithi169=b;count:2 2023-12-16T07:35:52.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:52 smithi007 bash[25009]: cluster 2023-12-16T07:35:51.192092+0000 mgr.y (mgr.24367) 3 : cluster [DBG] pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:52.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:52 smithi007 bash[25009]: audit 2023-12-16T07:35:51.195362+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:52 smithi169 bash[25083]: cluster 2023-12-16T07:35:51.182421+0000 mon.a (mon.0) 618 : cluster [DBG] mgrmap e19: y(active, since 1.86764s), standbys: x 2023-12-16T07:35:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:52 smithi169 bash[25083]: cephadm 2023-12-16T07:35:51.191999+0000 mgr.y (mgr.24367) 2 : cephadm [INF] Saving service node-exporter spec with placement smithi007=a;smithi169=b;count:2 2023-12-16T07:35:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:52 smithi169 bash[25083]: cluster 2023-12-16T07:35:51.192092+0000 mgr.y (mgr.24367) 3 : cluster [DBG] pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:52 smithi169 bash[25083]: audit 2023-12-16T07:35:51.195362+0000 mon.a (mon.0) 619 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:52.444 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:35:52.816 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:52 smithi007 bash[25282]: [16/Dec/2023:07:35:52] ENGINE Bus STARTING 2023-12-16T07:35:52.816 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:52 smithi007 bash[25282]: [16/Dec/2023:07:35:52] ENGINE Serving on https://172.21.15.7:7150 2023-12-16T07:35:52.816 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:35:52 smithi007 bash[25282]: [16/Dec/2023:07:35:52] ENGINE Bus STARTED 2023-12-16T07:35:53.215 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:53 smithi169 bash[25083]: cluster 2023-12-16T07:35:52.180749+0000 mgr.y (mgr.24367) 4 : cluster [DBG] pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:53.215 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:53 smithi169 bash[25083]: cluster 2023-12-16T07:35:52.206194+0000 mon.a (mon.0) 620 : cluster [DBG] mgrmap e20: y(active, since 2s), standbys: x 2023-12-16T07:35:53.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:53 smithi007 bash[30436]: cluster 2023-12-16T07:35:52.180749+0000 mgr.y (mgr.24367) 4 : cluster [DBG] pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:53.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:53 smithi007 bash[30436]: cluster 2023-12-16T07:35:52.206194+0000 mon.a (mon.0) 620 : cluster [DBG] mgrmap e20: y(active, since 2s), standbys: x 2023-12-16T07:35:53.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:53 smithi007 bash[30436]: cephadm 2023-12-16T07:35:52.536825+0000 mgr.y (mgr.24367) 5 : cephadm [INF] [16/Dec/2023:07:35:52] ENGINE Bus STARTING 2023-12-16T07:35:53.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:53 smithi007 bash[30436]: cephadm 2023-12-16T07:35:52.664145+0000 mgr.y (mgr.24367) 6 : cephadm [INF] [16/Dec/2023:07:35:52] ENGINE Serving on https://172.21.15.7:7150 2023-12-16T07:35:53.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:53 smithi007 bash[30436]: cephadm 2023-12-16T07:35:52.664899+0000 mgr.y (mgr.24367) 7 : cephadm [INF] [16/Dec/2023:07:35:52] ENGINE Bus STARTED 2023-12-16T07:35:53.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:53 smithi007 bash[30436]: audit 2023-12-16T07:35:52.673809+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:53.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:53 smithi007 bash[25009]: cluster 2023-12-16T07:35:52.180749+0000 mgr.y (mgr.24367) 4 : cluster [DBG] pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:53.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:53 smithi007 bash[25009]: cluster 2023-12-16T07:35:52.206194+0000 mon.a (mon.0) 620 : cluster [DBG] mgrmap e20: y(active, since 2s), standbys: x 2023-12-16T07:35:53.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:53 smithi007 bash[25009]: cephadm 2023-12-16T07:35:52.536825+0000 mgr.y (mgr.24367) 5 : cephadm [INF] [16/Dec/2023:07:35:52] ENGINE Bus STARTING 2023-12-16T07:35:53.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:53 smithi007 bash[25009]: cephadm 2023-12-16T07:35:52.664145+0000 mgr.y (mgr.24367) 6 : cephadm [INF] [16/Dec/2023:07:35:52] ENGINE Serving on https://172.21.15.7:7150 2023-12-16T07:35:53.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:53 smithi007 bash[25009]: cephadm 2023-12-16T07:35:52.664899+0000 mgr.y (mgr.24367) 7 : cephadm [INF] [16/Dec/2023:07:35:52] ENGINE Bus STARTED 2023-12-16T07:35:53.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:53 smithi007 bash[25009]: audit 2023-12-16T07:35:52.673809+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:53.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:53 smithi169 bash[25083]: cephadm 2023-12-16T07:35:52.536825+0000 mgr.y (mgr.24367) 5 : cephadm [INF] [16/Dec/2023:07:35:52] ENGINE Bus STARTING 2023-12-16T07:35:53.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:53 smithi169 bash[25083]: cephadm 2023-12-16T07:35:52.664145+0000 mgr.y (mgr.24367) 6 : cephadm [INF] [16/Dec/2023:07:35:52] ENGINE Serving on https://172.21.15.7:7150 2023-12-16T07:35:53.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:53 smithi169 bash[25083]: cephadm 2023-12-16T07:35:52.664899+0000 mgr.y (mgr.24367) 7 : cephadm [INF] [16/Dec/2023:07:35:52] ENGINE Bus STARTED 2023-12-16T07:35:53.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:53 smithi169 bash[25083]: audit 2023-12-16T07:35:52.673809+0000 mon.a (mon.0) 621 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:54.849 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled alertmanager update... 2023-12-16T07:35:55.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:55 smithi169 bash[25083]: audit 2023-12-16T07:35:54.064209+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:55 smithi169 bash[25083]: audit 2023-12-16T07:35:54.089216+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:55 smithi169 bash[25083]: audit 2023-12-16T07:35:54.558326+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:55 smithi169 bash[25083]: audit 2023-12-16T07:35:54.563685+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:55 smithi169 bash[25083]: audit 2023-12-16T07:35:54.793012+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:55 smithi169 bash[25083]: audit 2023-12-16T07:35:54.806898+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:55 smithi169 bash[25083]: audit 2023-12-16T07:35:54.813784+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:55 smithi169 bash[25083]: audit 2023-12-16T07:35:54.847906+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[25009]: audit 2023-12-16T07:35:54.064209+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[25009]: audit 2023-12-16T07:35:54.089216+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[25009]: audit 2023-12-16T07:35:54.558326+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[25009]: audit 2023-12-16T07:35:54.563685+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[25009]: audit 2023-12-16T07:35:54.793012+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[25009]: audit 2023-12-16T07:35:54.806898+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[25009]: audit 2023-12-16T07:35:54.813784+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[25009]: audit 2023-12-16T07:35:54.847906+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[30436]: audit 2023-12-16T07:35:54.064209+0000 mon.a (mon.0) 622 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[30436]: audit 2023-12-16T07:35:54.089216+0000 mon.a (mon.0) 623 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[30436]: audit 2023-12-16T07:35:54.558326+0000 mon.a (mon.0) 624 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[30436]: audit 2023-12-16T07:35:54.563685+0000 mon.a (mon.0) 625 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[30436]: audit 2023-12-16T07:35:54.793012+0000 mon.a (mon.0) 626 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[30436]: audit 2023-12-16T07:35:54.806898+0000 mon.a (mon.0) 627 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[30436]: audit 2023-12-16T07:35:54.813784+0000 mon.a (mon.0) 628 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:55 smithi007 bash[30436]: audit 2023-12-16T07:35:54.847906+0000 mon.a (mon.0) 629 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:35:55.740 DEBUG:teuthology.orchestra.run.smithi007:alertmanager.a> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@alertmanager.a.service 2023-12-16T07:35:55.742 INFO:tasks.cephadm:Adding grafana.a on smithi169 2023-12-16T07:35:55.742 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph orch apply grafana '1;smithi169=a' 2023-12-16T07:35:55.809 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:35:56.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[25009]: cluster 2023-12-16T07:35:54.181384+0000 mgr.y (mgr.24367) 8 : cluster [DBG] pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:56.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[25009]: cephadm 2023-12-16T07:35:54.560382+0000 mgr.y (mgr.24367) 9 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:35:56.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[25009]: cephadm 2023-12-16T07:35:54.566124+0000 mgr.y (mgr.24367) 10 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:35:56.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[25009]: cephadm 2023-12-16T07:35:54.673234+0000 mgr.y (mgr.24367) 11 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:35:56.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[25009]: cephadm 2023-12-16T07:35:54.682175+0000 mgr.y (mgr.24367) 12 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:35:56.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[25009]: cephadm 2023-12-16T07:35:54.839166+0000 mgr.y (mgr.24367) 13 : cephadm [INF] Deploying daemon prometheus.a on smithi169 2023-12-16T07:35:56.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[25009]: audit 2023-12-16T07:35:54.841073+0000 mgr.y (mgr.24367) 14 : audit [DBG] from='client.24386 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:35:56.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[25009]: cephadm 2023-12-16T07:35:54.843276+0000 mgr.y (mgr.24367) 15 : cephadm [INF] Saving service alertmanager spec with placement smithi007=a;count:1 2023-12-16T07:35:56.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[30436]: cluster 2023-12-16T07:35:54.181384+0000 mgr.y (mgr.24367) 8 : cluster [DBG] pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:56.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[30436]: cephadm 2023-12-16T07:35:54.560382+0000 mgr.y (mgr.24367) 9 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:35:56.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[30436]: cephadm 2023-12-16T07:35:54.566124+0000 mgr.y (mgr.24367) 10 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:35:56.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[30436]: cephadm 2023-12-16T07:35:54.673234+0000 mgr.y (mgr.24367) 11 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:35:56.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[30436]: cephadm 2023-12-16T07:35:54.682175+0000 mgr.y (mgr.24367) 12 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:35:56.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[30436]: cephadm 2023-12-16T07:35:54.839166+0000 mgr.y (mgr.24367) 13 : cephadm [INF] Deploying daemon prometheus.a on smithi169 2023-12-16T07:35:56.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[30436]: audit 2023-12-16T07:35:54.841073+0000 mgr.y (mgr.24367) 14 : audit [DBG] from='client.24386 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:35:56.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:56 smithi007 bash[30436]: cephadm 2023-12-16T07:35:54.843276+0000 mgr.y (mgr.24367) 15 : cephadm [INF] Saving service alertmanager spec with placement smithi007=a;count:1 2023-12-16T07:35:56.436 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:56 smithi169 bash[25083]: cluster 2023-12-16T07:35:54.181384+0000 mgr.y (mgr.24367) 8 : cluster [DBG] pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:56.437 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:56 smithi169 bash[25083]: cephadm 2023-12-16T07:35:54.560382+0000 mgr.y (mgr.24367) 9 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:35:56.437 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:56 smithi169 bash[25083]: cephadm 2023-12-16T07:35:54.566124+0000 mgr.y (mgr.24367) 10 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:35:56.437 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:56 smithi169 bash[25083]: cephadm 2023-12-16T07:35:54.673234+0000 mgr.y (mgr.24367) 11 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:35:56.437 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:56 smithi169 bash[25083]: cephadm 2023-12-16T07:35:54.682175+0000 mgr.y (mgr.24367) 12 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:35:56.437 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:56 smithi169 bash[25083]: cephadm 2023-12-16T07:35:54.839166+0000 mgr.y (mgr.24367) 13 : cephadm [INF] Deploying daemon prometheus.a on smithi169 2023-12-16T07:35:56.437 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:56 smithi169 bash[25083]: audit 2023-12-16T07:35:54.841073+0000 mgr.y (mgr.24367) 14 : audit [DBG] from='client.24386 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:35:56.437 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:56 smithi169 bash[25083]: cephadm 2023-12-16T07:35:54.843276+0000 mgr.y (mgr.24367) 15 : cephadm [INF] Saving service alertmanager spec with placement smithi007=a;count:1 2023-12-16T07:35:58.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:58 smithi169 bash[25083]: cluster 2023-12-16T07:35:56.182034+0000 mgr.y (mgr.24367) 16 : cluster [DBG] pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:58.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:58 smithi007 bash[25009]: cluster 2023-12-16T07:35:56.182034+0000 mgr.y (mgr.24367) 16 : cluster [DBG] pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:58.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:58 smithi007 bash[30436]: cluster 2023-12-16T07:35:56.182034+0000 mgr.y (mgr.24367) 16 : cluster [DBG] pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:59.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:35:59 smithi169 bash[25083]: cluster 2023-12-16T07:35:58.182768+0000 mgr.y (mgr.24367) 17 : cluster [DBG] pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:59.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:35:59 smithi007 bash[25009]: cluster 2023-12-16T07:35:58.182768+0000 mgr.y (mgr.24367) 17 : cluster [DBG] pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:35:59.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:35:59 smithi007 bash[30436]: cluster 2023-12-16T07:35:58.182768+0000 mgr.y (mgr.24367) 17 : cluster [DBG] pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:01.033 INFO:teuthology.orchestra.run.smithi169.stdout:Scheduled grafana update... 2023-12-16T07:36:02.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:02 smithi007 bash[30436]: cluster 2023-12-16T07:36:00.183372+0000 mgr.y (mgr.24367) 18 : cluster [DBG] pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:02.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:02 smithi007 bash[30436]: audit 2023-12-16T07:36:01.021088+0000 mgr.y (mgr.24367) 19 : audit [DBG] from='client.24392 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi169=a", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:36:02.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:02 smithi007 bash[30436]: cephadm 2023-12-16T07:36:01.024398+0000 mgr.y (mgr.24367) 20 : cephadm [INF] Saving service grafana spec with placement smithi169=a;count:1 2023-12-16T07:36:02.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:02 smithi007 bash[30436]: audit 2023-12-16T07:36:01.031488+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:02.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:02 smithi007 bash[25009]: cluster 2023-12-16T07:36:00.183372+0000 mgr.y (mgr.24367) 18 : cluster [DBG] pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:02.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:02 smithi007 bash[25009]: audit 2023-12-16T07:36:01.021088+0000 mgr.y (mgr.24367) 19 : audit [DBG] from='client.24392 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi169=a", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:36:02.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:02 smithi007 bash[25009]: cephadm 2023-12-16T07:36:01.024398+0000 mgr.y (mgr.24367) 20 : cephadm [INF] Saving service grafana spec with placement smithi169=a;count:1 2023-12-16T07:36:02.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:02 smithi007 bash[25009]: audit 2023-12-16T07:36:01.031488+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:02.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:02 smithi169 bash[25083]: cluster 2023-12-16T07:36:00.183372+0000 mgr.y (mgr.24367) 18 : cluster [DBG] pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:02.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:02 smithi169 bash[25083]: audit 2023-12-16T07:36:01.021088+0000 mgr.y (mgr.24367) 19 : audit [DBG] from='client.24392 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi169=a", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:36:02.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:02 smithi169 bash[25083]: cephadm 2023-12-16T07:36:01.024398+0000 mgr.y (mgr.24367) 20 : cephadm [INF] Saving service grafana spec with placement smithi169=a;count:1 2023-12-16T07:36:02.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:02 smithi169 bash[25083]: audit 2023-12-16T07:36:01.031488+0000 mon.a (mon.0) 630 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:02.782 DEBUG:teuthology.orchestra.run.smithi169:grafana.a> sudo journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@grafana.a.service 2023-12-16T07:36:02.784 INFO:tasks.cephadm:Setting up client nodes... 2023-12-16T07:36:02.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-12-16T07:36:02.885 INFO:journalctl@ceph.grafana.a.smithi169.stdout:-- Logs begin at Sun 2022-08-14 12:20:24 UTC. -- 2023-12-16T07:36:03.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:03 smithi007 bash[25009]: cluster 2023-12-16T07:36:02.184052+0000 mgr.y (mgr.24367) 21 : cluster [DBG] pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:03.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:03 smithi007 bash[30436]: cluster 2023-12-16T07:36:02.184052+0000 mgr.y (mgr.24367) 21 : cluster [DBG] pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:03.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:03 smithi169 bash[25083]: cluster 2023-12-16T07:36:02.184052+0000 mgr.y (mgr.24367) 21 : cluster [DBG] pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:05.564 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:05 smithi169 bash[25083]: cluster 2023-12-16T07:36:04.184818+0000 mgr.y (mgr.24367) 22 : cluster [DBG] pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:05.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:05 smithi007 bash[30436]: cluster 2023-12-16T07:36:04.184818+0000 mgr.y (mgr.24367) 22 : cluster [DBG] pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:05.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:05 smithi007 bash[25009]: cluster 2023-12-16T07:36:04.184818+0000 mgr.y (mgr.24367) 22 : cluster [DBG] pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:06.090 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2023-12-16T07:36:06.090 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBmU31lVvzrBBAA3T6wadEI9Iop0nkJZdnt1w== 2023-12-16T07:36:06.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:06 smithi169 bash[25083]: audit 2023-12-16T07:36:06.081371+0000 mon.c (mon.1) 11 : audit [INF] from='client.? 172.21.15.7:0/3624896838' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T07:36:06.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:06 smithi169 bash[25083]: audit 2023-12-16T07:36:06.082171+0000 mon.a (mon.0) 631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T07:36:06.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:06 smithi169 bash[25083]: audit 2023-12-16T07:36:06.087125+0000 mon.a (mon.0) 632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-16T07:36:06.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:06 smithi007 bash[25009]: audit 2023-12-16T07:36:06.081371+0000 mon.c (mon.1) 11 : audit [INF] from='client.? 172.21.15.7:0/3624896838' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T07:36:06.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:06 smithi007 bash[25009]: audit 2023-12-16T07:36:06.082171+0000 mon.a (mon.0) 631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T07:36:06.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:06 smithi007 bash[25009]: audit 2023-12-16T07:36:06.087125+0000 mon.a (mon.0) 632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-16T07:36:06.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:06 smithi007 bash[30436]: audit 2023-12-16T07:36:06.081371+0000 mon.c (mon.1) 11 : audit [INF] from='client.? 172.21.15.7:0/3624896838' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T07:36:06.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:06 smithi007 bash[30436]: audit 2023-12-16T07:36:06.082171+0000 mon.a (mon.0) 631 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T07:36:06.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:06 smithi007 bash[30436]: audit 2023-12-16T07:36:06.087125+0000 mon.a (mon.0) 632 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-16T07:36:06.818 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-12-16T07:36:06.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-12-16T07:36:06.819 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-16T07:36:06.841 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-12-16T07:36:07.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:07 smithi169 bash[25083]: cluster 2023-12-16T07:36:06.185614+0000 mgr.y (mgr.24367) 23 : cluster [DBG] pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:07.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:07 smithi007 bash[25009]: cluster 2023-12-16T07:36:06.185614+0000 mgr.y (mgr.24367) 23 : cluster [DBG] pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:07.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:07 smithi007 bash[30436]: cluster 2023-12-16T07:36:06.185614+0000 mgr.y (mgr.24367) 23 : cluster [DBG] pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:08.892 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:08 smithi169 systemd[1]: Started Ceph prometheus.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:36:09.473 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:09 smithi007 systemd[1]: Started Ceph node-exporter.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:36:09.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:09 smithi007 bash[25009]: cluster 2023-12-16T07:36:08.186410+0000 mgr.y (mgr.24367) 24 : cluster [DBG] pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:09.724 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:09 smithi007 bash[25009]: audit 2023-12-16T07:36:08.723847+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:09.724 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:09 smithi007 bash[45201]: Unable to find image 'quay.io/prometheus/node-exporter:v1.3.1' locally 2023-12-16T07:36:09.885 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:09 smithi169 bash[25083]: cluster 2023-12-16T07:36:08.186410+0000 mgr.y (mgr.24367) 24 : cluster [DBG] pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:09.885 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:09 smithi169 bash[25083]: audit 2023-12-16T07:36:08.723847+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:10.006 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:09 smithi007 bash[25009]: cephadm 2023-12-16T07:36:08.728525+0000 mgr.y (mgr.24367) 25 : cephadm [INF] Deploying daemon node-exporter.a on smithi007 2023-12-16T07:36:10.006 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:09 smithi007 bash[25009]: audit 2023-12-16T07:36:09.481812+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:10.007 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:09 smithi007 bash[30436]: cluster 2023-12-16T07:36:08.186410+0000 mgr.y (mgr.24367) 24 : cluster [DBG] pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:10.007 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:09 smithi007 bash[30436]: audit 2023-12-16T07:36:08.723847+0000 mon.a (mon.0) 633 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:10.007 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:09 smithi007 bash[30436]: cephadm 2023-12-16T07:36:08.728525+0000 mgr.y (mgr.24367) 25 : cephadm [INF] Deploying daemon node-exporter.a on smithi007 2023-12-16T07:36:10.007 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:09 smithi007 bash[30436]: audit 2023-12-16T07:36:09.481812+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:10.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:09 smithi169 bash[25083]: cephadm 2023-12-16T07:36:08.728525+0000 mgr.y (mgr.24367) 25 : cephadm [INF] Deploying daemon node-exporter.a on smithi007 2023-12-16T07:36:10.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:09 smithi169 bash[25083]: audit 2023-12-16T07:36:09.481812+0000 mon.a (mon.0) 634 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:10.316 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[45201]: v1.3.1: Pulling from prometheus/node-exporter 2023-12-16T07:36:10.316 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[45201]: aa2a8d90b84c: Pulling fs layer 2023-12-16T07:36:10.316 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[45201]: b45d31ee2d7f: Pulling fs layer 2023-12-16T07:36:10.316 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[45201]: b5db1e299295: Pulling fs layer 2023-12-16T07:36:10.683 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:10 smithi169 systemd[1]: Started Ceph node-exporter.b for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:36:10.683 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40213]: Unable to find image 'quay.io/prometheus/node-exporter:v1.3.1' locally 2023-12-16T07:36:10.731 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[45201]: b45d31ee2d7f: Download complete 2023-12-16T07:36:10.731 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[45201]: aa2a8d90b84c: Verifying Checksum 2023-12-16T07:36:10.731 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[45201]: aa2a8d90b84c: Download complete 2023-12-16T07:36:10.731 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[45201]: b5db1e299295: Verifying Checksum 2023-12-16T07:36:10.731 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[45201]: b5db1e299295: Download complete 2023-12-16T07:36:10.731 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[45201]: aa2a8d90b84c: Pull complete 2023-12-16T07:36:11.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[25009]: cephadm 2023-12-16T07:36:09.492169+0000 mgr.y (mgr.24367) 26 : cephadm [INF] Deploying daemon node-exporter.b on smithi169 2023-12-16T07:36:11.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[25009]: audit 2023-12-16T07:36:10.267673+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:11.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[25009]: audit 2023-12-16T07:36:10.386078+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:11.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[25009]: audit 2023-12-16T07:36:10.388255+0000 mon.a (mon.0) 637 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:36:11.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[25009]: audit 2023-12-16T07:36:10.390316+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:36:11.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[25009]: audit 2023-12-16T07:36:10.391643+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:36:11.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[30436]: cephadm 2023-12-16T07:36:09.492169+0000 mgr.y (mgr.24367) 26 : cephadm [INF] Deploying daemon node-exporter.b on smithi169 2023-12-16T07:36:11.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[30436]: audit 2023-12-16T07:36:10.267673+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:11.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[30436]: audit 2023-12-16T07:36:10.386078+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:11.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[30436]: audit 2023-12-16T07:36:10.388255+0000 mon.a (mon.0) 637 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:36:11.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[30436]: audit 2023-12-16T07:36:10.390316+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:36:11.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:10 smithi007 bash[30436]: audit 2023-12-16T07:36:10.391643+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:36:11.114 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[25083]: cephadm 2023-12-16T07:36:09.492169+0000 mgr.y (mgr.24367) 26 : cephadm [INF] Deploying daemon node-exporter.b on smithi169 2023-12-16T07:36:11.115 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[25083]: audit 2023-12-16T07:36:10.267673+0000 mon.a (mon.0) 635 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:11.115 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[25083]: audit 2023-12-16T07:36:10.386078+0000 mon.a (mon.0) 636 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:11.115 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[25083]: audit 2023-12-16T07:36:10.388255+0000 mon.a (mon.0) 637 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:36:11.115 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[25083]: audit 2023-12-16T07:36:10.390316+0000 mon.a (mon.0) 638 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:36:11.115 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[25083]: audit 2023-12-16T07:36:10.391643+0000 mon.a (mon.0) 639 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:36:11.115 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.923Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2023-12-16T07:36:11.115 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.923Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2023-12-16T07:36:11.115 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.923Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2023-12-16T07:36:11.115 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.923Z caller=main.go:518 level=info host_details="(Linux 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 smithi169 (none))" 2023-12-16T07:36:11.115 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.923Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2023-12-16T07:36:11.115 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.923Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2023-12-16T07:36:11.116 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.925Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2023-12-16T07:36:11.116 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.926Z caller=main.go:923 level=info msg="Starting TSDB ..." 2023-12-16T07:36:11.116 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.929Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2023-12-16T07:36:11.116 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.931Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2023-12-16T07:36:11.116 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.931Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.195µs 2023-12-16T07:36:11.116 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.931Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2023-12-16T07:36:11.116 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.931Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2023-12-16T07:36:11.116 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.931Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.273µs wal_replay_duration=278.231µs total_replay_duration=335.285µs 2023-12-16T07:36:11.116 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.932Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2023-12-16T07:36:11.116 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.932Z caller=main.go:947 level=info msg="TSDB started" 2023-12-16T07:36:11.116 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.932Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-12-16T07:36:11.116 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.959Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=26.483949ms db_storage=966ns remote_storage=1.639µs web_handler=317ns query_engine=983ns scrape=342.186µs scrape_sd=20.911µs notify=1.104µs notify_sd=1.887µs rules=25.871612ms 2023-12-16T07:36:11.116 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:36:10 smithi169 bash[40088]: ts=2023-12-16T07:36:10.959Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2023-12-16T07:36:11.117 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:11 smithi169 bash[40213]: v1.3.1: Pulling from prometheus/node-exporter 2023-12-16T07:36:11.432 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:11 smithi169 bash[40213]: aa2a8d90b84c: Pulling fs layer 2023-12-16T07:36:11.433 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:11 smithi169 bash[40213]: b45d31ee2d7f: Pulling fs layer 2023-12-16T07:36:11.433 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:11 smithi169 bash[40213]: b5db1e299295: Pulling fs layer 2023-12-16T07:36:11.540 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:11 smithi007 bash[45201]: b45d31ee2d7f: Pull complete 2023-12-16T07:36:11.689 INFO:teuthology.orchestra.run.smithi169.stdout:[client.1] 2023-12-16T07:36:11.689 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBrU31lKTuWKBAAwH9xCf8i6IjObQTuly334w== 2023-12-16T07:36:11.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:11 smithi007 bash[30436]: cluster 2023-12-16T07:36:10.187074+0000 mgr.y (mgr.24367) 27 : cluster [DBG] pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:11.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:11 smithi007 bash[30436]: audit 2023-12-16T07:36:11.680410+0000 mon.b (mon.2) 24 : audit [INF] from='client.? 172.21.15.169:0/3264920711' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T07:36:11.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:11 smithi007 bash[30436]: audit 2023-12-16T07:36:11.680517+0000 mon.a (mon.0) 640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T07:36:11.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:11 smithi007 bash[30436]: audit 2023-12-16T07:36:11.688232+0000 mon.a (mon.0) 641 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-16T07:36:11.817 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:11 smithi007 bash[45201]: b5db1e299295: Pull complete 2023-12-16T07:36:11.817 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:11 smithi007 bash[45201]: Digest: sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd 2023-12-16T07:36:11.817 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:11 smithi007 bash[45201]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.3.1 2023-12-16T07:36:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:11 smithi007 bash[25009]: cluster 2023-12-16T07:36:10.187074+0000 mgr.y (mgr.24367) 27 : cluster [DBG] pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:11 smithi007 bash[25009]: audit 2023-12-16T07:36:11.680410+0000 mon.b (mon.2) 24 : audit [INF] from='client.? 172.21.15.169:0/3264920711' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T07:36:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:11 smithi007 bash[25009]: audit 2023-12-16T07:36:11.680517+0000 mon.a (mon.0) 640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T07:36:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:11 smithi007 bash[25009]: audit 2023-12-16T07:36:11.688232+0000 mon.a (mon.0) 641 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-16T07:36:11.921 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:11 smithi169 bash[25083]: cluster 2023-12-16T07:36:10.187074+0000 mgr.y (mgr.24367) 27 : cluster [DBG] pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:11.921 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:11 smithi169 bash[25083]: audit 2023-12-16T07:36:11.680410+0000 mon.b (mon.2) 24 : audit [INF] from='client.? 172.21.15.169:0/3264920711' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T07:36:11.921 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:11 smithi169 bash[25083]: audit 2023-12-16T07:36:11.680517+0000 mon.a (mon.0) 640 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-12-16T07:36:11.921 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:11 smithi169 bash[25083]: audit 2023-12-16T07:36:11.688232+0000 mon.a (mon.0) 641 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-12-16T07:36:11.922 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:11 smithi169 bash[40213]: b45d31ee2d7f: Download complete 2023-12-16T07:36:11.922 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:11 smithi169 bash[40213]: aa2a8d90b84c: Verifying Checksum 2023-12-16T07:36:11.922 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:11 smithi169 bash[40213]: aa2a8d90b84c: Download complete 2023-12-16T07:36:11.922 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:11 smithi169 bash[40213]: b5db1e299295: Verifying Checksum 2023-12-16T07:36:11.922 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:11 smithi169 bash[40213]: b5db1e299295: Download complete 2023-12-16T07:36:11.922 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:11 smithi169 bash[40213]: aa2a8d90b84c: Pull complete 2023-12-16T07:36:12.729 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2023-12-16T07:36:12.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2023-12-16T07:36:12.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-16T07:36:12.753 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-16T07:36:12.753 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-12-16T07:36:12.754 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph mgr dump --format=json 2023-12-16T07:36:12.933 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:12 smithi169 bash[40213]: b45d31ee2d7f: Pull complete 2023-12-16T07:36:13.296 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:13 smithi169 bash[25083]: cluster 2023-12-16T07:36:12.187868+0000 mgr.y (mgr.24367) 28 : cluster [DBG] pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:13.297 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:13 smithi169 bash[40213]: b5db1e299295: Pull complete 2023-12-16T07:36:13.297 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:13 smithi169 bash[40213]: Digest: sha256:f2269e73124dd0f60a7d19a2ce1264d33d08a985aed0ee6b0b89d0be470592cd 2023-12-16T07:36:13.297 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:13 smithi169 bash[40213]: Status: Downloaded newer image for quay.io/prometheus/node-exporter:v1.3.1 2023-12-16T07:36:13.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[30436]: cluster 2023-12-16T07:36:12.187868+0000 mgr.y (mgr.24367) 28 : cluster [DBG] pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:13.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[25009]: cluster 2023-12-16T07:36:12.187868+0000 mgr.y (mgr.24367) 28 : cluster [DBG] pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:13.566 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.204Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2023-12-16T07:36:13.566 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.204Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2023-12-16T07:36:13.566 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.205Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2023-12-16T07:36:13.567 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.205Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2023-12-16T07:36:13.567 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2023-12-16T07:36:13.567 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=arp 2023-12-16T07:36:13.567 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=bcache 2023-12-16T07:36:13.567 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=bonding 2023-12-16T07:36:13.567 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=btrfs 2023-12-16T07:36:13.567 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=conntrack 2023-12-16T07:36:13.567 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=cpu 2023-12-16T07:36:13.567 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=cpufreq 2023-12-16T07:36:13.567 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=diskstats 2023-12-16T07:36:13.567 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=dmi 2023-12-16T07:36:13.567 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=edac 2023-12-16T07:36:13.567 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=entropy 2023-12-16T07:36:13.567 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=fibrechannel 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=filefd 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=filesystem 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=hwmon 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=infiniband 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.206Z caller=node_exporter.go:115 level=info collector=ipvs 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=loadavg 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=mdadm 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=meminfo 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=netclass 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=netdev 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=netstat 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=nfs 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=nfsd 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=nvme 2023-12-16T07:36:13.568 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=os 2023-12-16T07:36:13.569 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2023-12-16T07:36:13.569 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=pressure 2023-12-16T07:36:13.569 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=rapl 2023-12-16T07:36:13.569 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=schedstat 2023-12-16T07:36:13.569 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=sockstat 2023-12-16T07:36:13.569 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=softnet 2023-12-16T07:36:13.569 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=stat 2023-12-16T07:36:13.569 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=tapestats 2023-12-16T07:36:13.569 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=textfile 2023-12-16T07:36:13.569 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=thermal_zone 2023-12-16T07:36:13.569 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=time 2023-12-16T07:36:13.569 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=udp_queues 2023-12-16T07:36:13.569 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=uname 2023-12-16T07:36:13.569 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=vmstat 2023-12-16T07:36:13.570 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=xfs 2023-12-16T07:36:13.570 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:115 level=info collector=zfs 2023-12-16T07:36:13.570 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2023-12-16T07:36:13.570 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:36:13 smithi007 bash[45201]: ts=2023-12-16T07:36:13.207Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2023-12-16T07:36:14.683 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.518Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2023-12-16T07:36:14.683 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.518Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2023-12-16T07:36:14.684 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.519Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2023-12-16T07:36:14.684 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.519Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2023-12-16T07:36:14.684 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2023-12-16T07:36:14.684 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=arp 2023-12-16T07:36:14.684 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=bcache 2023-12-16T07:36:14.684 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=bonding 2023-12-16T07:36:14.684 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=btrfs 2023-12-16T07:36:14.684 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=conntrack 2023-12-16T07:36:14.684 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=cpu 2023-12-16T07:36:14.684 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=cpufreq 2023-12-16T07:36:14.684 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=diskstats 2023-12-16T07:36:14.684 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=dmi 2023-12-16T07:36:14.684 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=edac 2023-12-16T07:36:14.685 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=entropy 2023-12-16T07:36:14.685 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=fibrechannel 2023-12-16T07:36:14.685 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=filefd 2023-12-16T07:36:14.685 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=filesystem 2023-12-16T07:36:14.685 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=hwmon 2023-12-16T07:36:14.685 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=infiniband 2023-12-16T07:36:14.685 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=ipvs 2023-12-16T07:36:14.685 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=loadavg 2023-12-16T07:36:14.685 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=mdadm 2023-12-16T07:36:14.685 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=meminfo 2023-12-16T07:36:14.685 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=netclass 2023-12-16T07:36:14.685 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=netdev 2023-12-16T07:36:14.685 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=netstat 2023-12-16T07:36:14.685 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=nfs 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=nfsd 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=nvme 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=os 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=pressure 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=rapl 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=schedstat 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=sockstat 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=softnet 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=stat 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=tapestats 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=textfile 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=thermal_zone 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=time 2023-12-16T07:36:14.686 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=udp_queues 2023-12-16T07:36:14.687 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=uname 2023-12-16T07:36:14.687 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=vmstat 2023-12-16T07:36:14.687 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=xfs 2023-12-16T07:36:14.687 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:115 level=info collector=zfs 2023-12-16T07:36:14.687 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2023-12-16T07:36:14.687 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[40213]: ts=2023-12-16T07:36:14.520Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2023-12-16T07:36:15.182 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:15.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[25083]: audit 2023-12-16T07:36:13.855302+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:15.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[25083]: audit 2023-12-16T07:36:14.006829+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:15.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[25083]: audit 2023-12-16T07:36:14.016166+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:15.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:14 smithi169 bash[25083]: cephadm 2023-12-16T07:36:14.031922+0000 mgr.y (mgr.24367) 29 : cephadm [INF] Deploying daemon alertmanager.a on smithi007 2023-12-16T07:36:15.213 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:14 smithi007 bash[25009]: audit 2023-12-16T07:36:13.855302+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:15.213 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:14 smithi007 bash[25009]: audit 2023-12-16T07:36:14.006829+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:15.213 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:14 smithi007 bash[25009]: audit 2023-12-16T07:36:14.016166+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:15.213 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:14 smithi007 bash[25009]: cephadm 2023-12-16T07:36:14.031922+0000 mgr.y (mgr.24367) 29 : cephadm [INF] Deploying daemon alertmanager.a on smithi007 2023-12-16T07:36:15.213 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:14 smithi007 bash[30436]: audit 2023-12-16T07:36:13.855302+0000 mon.a (mon.0) 642 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:15.214 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:14 smithi007 bash[30436]: audit 2023-12-16T07:36:14.006829+0000 mon.a (mon.0) 643 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:15.214 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:14 smithi007 bash[30436]: audit 2023-12-16T07:36:14.016166+0000 mon.a (mon.0) 644 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:15.214 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:14 smithi007 bash[30436]: cephadm 2023-12-16T07:36:14.031922+0000 mgr.y (mgr.24367) 29 : cephadm [INF] Deploying daemon alertmanager.a on smithi007 2023-12-16T07:36:16.343 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:16 smithi007 bash[25009]: cluster 2023-12-16T07:36:14.188515+0000 mgr.y (mgr.24367) 30 : cluster [DBG] pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:16.343 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:16 smithi007 bash[25009]: audit 2023-12-16T07:36:15.276748+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:16.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:16 smithi007 bash[30436]: cluster 2023-12-16T07:36:14.188515+0000 mgr.y (mgr.24367) 30 : cluster [DBG] pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:16.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:16 smithi007 bash[30436]: audit 2023-12-16T07:36:15.276748+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:16.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:16 smithi169 bash[25083]: cluster 2023-12-16T07:36:14.188515+0000 mgr.y (mgr.24367) 30 : cluster [DBG] pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:16.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:16 smithi169 bash[25083]: audit 2023-12-16T07:36:15.276748+0000 mon.a (mon.0) 645 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:18.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:18 smithi007 bash[25009]: cluster 2023-12-16T07:36:16.189129+0000 mgr.y (mgr.24367) 31 : cluster [DBG] pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:18.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:18 smithi007 bash[30436]: cluster 2023-12-16T07:36:16.189129+0000 mgr.y (mgr.24367) 31 : cluster [DBG] pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:18.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:18 smithi169 bash[25083]: cluster 2023-12-16T07:36:16.189129+0000 mgr.y (mgr.24367) 31 : cluster [DBG] pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:20.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:20 smithi007 bash[30436]: cluster 2023-12-16T07:36:18.190008+0000 mgr.y (mgr.24367) 32 : cluster [DBG] pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:20.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:20 smithi007 bash[25009]: cluster 2023-12-16T07:36:18.190008+0000 mgr.y (mgr.24367) 32 : cluster [DBG] pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:20.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:20 smithi169 bash[25083]: cluster 2023-12-16T07:36:18.190008+0000 mgr.y (mgr.24367) 32 : cluster [DBG] pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:21.171 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-16T07:36:22.226 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:22 smithi007 bash[30436]: cluster 2023-12-16T07:36:20.190691+0000 mgr.y (mgr.24367) 33 : cluster [DBG] pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:22.226 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:22 smithi007 bash[30436]: audit 2023-12-16T07:36:21.160678+0000 mon.a (mon.0) 646 : audit [DBG] from='client.? 172.21.15.7:0/1248539438' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-12-16T07:36:22.227 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:22 smithi007 bash[25009]: cluster 2023-12-16T07:36:20.190691+0000 mgr.y (mgr.24367) 33 : cluster [DBG] pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:22.227 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:22 smithi007 bash[25009]: audit 2023-12-16T07:36:21.160678+0000 mon.a (mon.0) 646 : audit [DBG] from='client.? 172.21.15.7:0/1248539438' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-12-16T07:36:22.313 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"active_gid":24367,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":942562873},{"type":"v1","addr":"172.21.15.7:6801","nonce":942562873}]},"active_addr":"172.21.15.7:6801/942562873","active_change":"2023-12-16T07:35:49.314762+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24356,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.7:8443/","prometheus":"http://172.21.15.7:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":906851426}]},{"addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":1908420919}]},{"addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":2952790388}]},{"addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3262094376}]}]}} 2023-12-16T07:36:22.317 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-12-16T07:36:22.317 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-12-16T07:36:22.317 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-16T07:36:22.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:36:22 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:36:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:36:22.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:22 smithi169 bash[25083]: cluster 2023-12-16T07:36:20.190691+0000 mgr.y (mgr.24367) 33 : cluster [DBG] pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:22.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:22 smithi169 bash[25083]: audit 2023-12-16T07:36:21.160678+0000 mon.a (mon.0) 646 : audit [DBG] from='client.? 172.21.15.7:0/1248539438' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-12-16T07:36:23.375 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:23 smithi007 bash[30436]: cluster 2023-12-16T07:36:22.191314+0000 mgr.y (mgr.24367) 34 : cluster [DBG] pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:23.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:23 smithi007 bash[25009]: cluster 2023-12-16T07:36:22.191314+0000 mgr.y (mgr.24367) 34 : cluster [DBG] pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:23 smithi169 bash[25083]: cluster 2023-12-16T07:36:22.191314+0000 mgr.y (mgr.24367) 34 : cluster [DBG] pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:24.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:36:23 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:36:23] "GET /metrics HTTP/1.1" 200 191505 "" "Prometheus/2.33.4" 2023-12-16T07:36:25.047 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:25.066 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:36:24 smithi007 systemd[1]: Started Ceph alertmanager.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:36:26.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[25009]: cluster 2023-12-16T07:36:24.192069+0000 mgr.y (mgr.24367) 35 : cluster [DBG] pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:26.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[25009]: audit 2023-12-16T07:36:24.806018+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[25009]: audit 2023-12-16T07:36:24.952367+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[25009]: audit 2023-12-16T07:36:24.959091+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[25009]: audit 2023-12-16T07:36:24.963632+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T07:36:26.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[25009]: audit 2023-12-16T07:36:24.964445+0000 mgr.y (mgr.24367) 36 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T07:36:26.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[25009]: audit 2023-12-16T07:36:24.971255+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[25009]: cephadm 2023-12-16T07:36:24.981767+0000 mgr.y (mgr.24367) 37 : cephadm [INF] Deploying daemon grafana.a on smithi169 2023-12-16T07:36:26.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[25009]: audit 2023-12-16T07:36:25.287493+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[30436]: cluster 2023-12-16T07:36:24.192069+0000 mgr.y (mgr.24367) 35 : cluster [DBG] pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:26.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[30436]: audit 2023-12-16T07:36:24.806018+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[30436]: audit 2023-12-16T07:36:24.952367+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[30436]: audit 2023-12-16T07:36:24.959091+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[30436]: audit 2023-12-16T07:36:24.963632+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T07:36:26.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[30436]: audit 2023-12-16T07:36:24.964445+0000 mgr.y (mgr.24367) 36 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T07:36:26.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[30436]: audit 2023-12-16T07:36:24.971255+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[30436]: cephadm 2023-12-16T07:36:24.981767+0000 mgr.y (mgr.24367) 37 : cephadm [INF] Deploying daemon grafana.a on smithi169 2023-12-16T07:36:26.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:25 smithi007 bash[30436]: audit 2023-12-16T07:36:25.287493+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:25 smithi169 bash[25083]: cluster 2023-12-16T07:36:24.192069+0000 mgr.y (mgr.24367) 35 : cluster [DBG] pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:26.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:25 smithi169 bash[25083]: audit 2023-12-16T07:36:24.806018+0000 mon.a (mon.0) 647 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:25 smithi169 bash[25083]: audit 2023-12-16T07:36:24.952367+0000 mon.a (mon.0) 648 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:25 smithi169 bash[25083]: audit 2023-12-16T07:36:24.959091+0000 mon.a (mon.0) 649 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:25 smithi169 bash[25083]: audit 2023-12-16T07:36:24.963632+0000 mon.a (mon.0) 650 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T07:36:26.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:25 smithi169 bash[25083]: audit 2023-12-16T07:36:24.964445+0000 mgr.y (mgr.24367) 36 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T07:36:26.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:25 smithi169 bash[25083]: audit 2023-12-16T07:36:24.971255+0000 mon.a (mon.0) 651 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:25 smithi169 bash[25083]: cephadm 2023-12-16T07:36:24.981767+0000 mgr.y (mgr.24367) 37 : cephadm [INF] Deploying daemon grafana.a on smithi169 2023-12-16T07:36:26.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:25 smithi169 bash[25083]: audit 2023-12-16T07:36:25.287493+0000 mon.a (mon.0) 652 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:26.798 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:36:26 smithi007 bash[45990]: level=info ts=2023-12-16T07:36:26.649Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2023-12-16T07:36:26.798 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:36:26 smithi007 bash[45990]: level=info ts=2023-12-16T07:36:26.649Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2023-12-16T07:36:26.799 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:36:26 smithi007 bash[45990]: level=info ts=2023-12-16T07:36:26.652Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.7 port=9094 2023-12-16T07:36:26.799 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:36:26 smithi007 bash[45990]: level=info ts=2023-12-16T07:36:26.654Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2023-12-16T07:36:26.799 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:36:26 smithi007 bash[45990]: level=info ts=2023-12-16T07:36:26.705Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2023-12-16T07:36:26.799 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:36:26 smithi007 bash[45990]: level=info ts=2023-12-16T07:36:26.705Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2023-12-16T07:36:26.799 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:36:26 smithi007 bash[45990]: level=info ts=2023-12-16T07:36:26.710Z caller=main.go:518 msg=Listening address=:9093 2023-12-16T07:36:26.799 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:36:26 smithi007 bash[45990]: level=info ts=2023-12-16T07:36:26.710Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2023-12-16T07:36:28.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:27 smithi007 bash[25009]: cluster 2023-12-16T07:36:26.192748+0000 mgr.y (mgr.24367) 38 : cluster [DBG] pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:28.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:27 smithi007 bash[30436]: cluster 2023-12-16T07:36:26.192748+0000 mgr.y (mgr.24367) 38 : cluster [DBG] pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:28.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:27 smithi169 bash[25083]: cluster 2023-12-16T07:36:26.192748+0000 mgr.y (mgr.24367) 38 : cluster [DBG] pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:28.816 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:36:28 smithi007 bash[45990]: level=info ts=2023-12-16T07:36:28.655Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001016833s 2023-12-16T07:36:28.856 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-16T07:36:28.857 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":59,"fsid":"8b71d3dc-9be4-11ee-95a5-87774f69a715","created":"2023-12-16T07:28:10.358509+0000","modified":"2023-12-16T07:35:49.313881+0000","last_up_change":"2023-12-16T07:35:22.069655+0000","last_in_change":"2023-12-16T07:34:57.664166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T07:32:19.902470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d8aa3a96-fde6-48ee-9a0a-4ad9a0d4f427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3002763011},{"type":"v1","addr":"172.21.15.7:6803","nonce":3002763011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3002763011},{"type":"v1","addr":"172.21.15.7:6805","nonce":3002763011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3002763011},{"type":"v1","addr":"172.21.15.7:6809","nonce":3002763011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3002763011},{"type":"v1","addr":"172.21.15.7:6807","nonce":3002763011}]},"public_addr":"172.21.15.7:6803/3002763011","cluster_addr":"172.21.15.7:6805/3002763011","heartbeat_back_addr":"172.21.15.7:6809/3002763011","heartbeat_front_addr":"172.21.15.7:6807/3002763011","state":["exists","up"]},{"osd":1,"uuid":"b52565e1-b236-4f11-8356-c6e90f476298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":4053186404},{"type":"v1","addr":"172.21.15.7:6811","nonce":4053186404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":4053186404},{"type":"v1","addr":"172.21.15.7:6813","nonce":4053186404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":4053186404},{"type":"v1","addr":"172.21.15.7:6817","nonce":4053186404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":4053186404},{"type":"v1","addr":"172.21.15.7:6815","nonce":4053186404}]},"public_addr":"172.21.15.7:6811/4053186404","cluster_addr":"172.21.15.7:6813/4053186404","heartbeat_back_addr":"172.21.15.7:6817/4053186404","heartbeat_front_addr":"172.21.15.7:6815/4053186404","state":["exists","up"]},{"osd":2,"uuid":"5a43c2e0-1837-4880-b904-5ea6041d45a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":2173677438},{"type":"v1","addr":"172.21.15.7:6819","nonce":2173677438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":2173677438},{"type":"v1","addr":"172.21.15.7:6821","nonce":2173677438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":2173677438},{"type":"v1","addr":"172.21.15.7:6825","nonce":2173677438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":2173677438},{"type":"v1","addr":"172.21.15.7:6823","nonce":2173677438}]},"public_addr":"172.21.15.7:6819/2173677438","cluster_addr":"172.21.15.7:6821/2173677438","heartbeat_back_addr":"172.21.15.7:6825/2173677438","heartbeat_front_addr":"172.21.15.7:6823/2173677438","state":["exists","up"]},{"osd":3,"uuid":"4776e261-fa8a-4652-a32f-83710dda5dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1623735309},{"type":"v1","addr":"172.21.15.7:6827","nonce":1623735309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1623735309},{"type":"v1","addr":"172.21.15.7:6829","nonce":1623735309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1623735309},{"type":"v1","addr":"172.21.15.7:6833","nonce":1623735309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1623735309},{"type":"v1","addr":"172.21.15.7:6831","nonce":1623735309}]},"public_addr":"172.21.15.7:6827/1623735309","cluster_addr":"172.21.15.7:6829/1623735309","heartbeat_back_addr":"172.21.15.7:6833/1623735309","heartbeat_front_addr":"172.21.15.7:6831/1623735309","state":["exists","up"]},{"osd":4,"uuid":"d5631b50-8179-4ed7-8f03-4de60d19bb69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3747189467},{"type":"v1","addr":"172.21.15.169:6801","nonce":3747189467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3747189467},{"type":"v1","addr":"172.21.15.169:6803","nonce":3747189467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3747189467},{"type":"v1","addr":"172.21.15.169:6807","nonce":3747189467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3747189467},{"type":"v1","addr":"172.21.15.169:6805","nonce":3747189467}]},"public_addr":"172.21.15.169:6801/3747189467","cluster_addr":"172.21.15.169:6803/3747189467","heartbeat_back_addr":"172.21.15.169:6807/3747189467","heartbeat_front_addr":"172.21.15.169:6805/3747189467","state":["exists","up"]},{"osd":5,"uuid":"70f322c0-b8b5-439c-bf2a-8688a28ca919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":3877143549},{"type":"v1","addr":"172.21.15.169:6809","nonce":3877143549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":3877143549},{"type":"v1","addr":"172.21.15.169:6811","nonce":3877143549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3877143549},{"type":"v1","addr":"172.21.15.169:6815","nonce":3877143549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":3877143549},{"type":"v1","addr":"172.21.15.169:6813","nonce":3877143549}]},"public_addr":"172.21.15.169:6809/3877143549","cluster_addr":"172.21.15.169:6811/3877143549","heartbeat_back_addr":"172.21.15.169:6815/3877143549","heartbeat_front_addr":"172.21.15.169:6813/3877143549","state":["exists","up"]},{"osd":6,"uuid":"75186dc4-13a9-4c53-9db6-07724e058c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":1373876707},{"type":"v1","addr":"172.21.15.169:6817","nonce":1373876707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":1373876707},{"type":"v1","addr":"172.21.15.169:6819","nonce":1373876707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":1373876707},{"type":"v1","addr":"172.21.15.169:6823","nonce":1373876707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":1373876707},{"type":"v1","addr":"172.21.15.169:6821","nonce":1373876707}]},"public_addr":"172.21.15.169:6817/1373876707","cluster_addr":"172.21.15.169:6819/1373876707","heartbeat_back_addr":"172.21.15.169:6823/1373876707","heartbeat_front_addr":"172.21.15.169:6821/1373876707","state":["exists","up"]},{"osd":7,"uuid":"1f760eeb-da8d-4fce-a8f4-cf42d952fcc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1772368148},{"type":"v1","addr":"172.21.15.169:6825","nonce":1772368148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1772368148},{"type":"v1","addr":"172.21.15.169:6827","nonce":1772368148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1772368148},{"type":"v1","addr":"172.21.15.169:6831","nonce":1772368148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1772368148},{"type":"v1","addr":"172.21.15.169:6829","nonce":1772368148}]},"public_addr":"172.21.15.169:6825/1772368148","cluster_addr":"172.21.15.169:6827/1772368148","heartbeat_back_addr":"172.21.15.169:6831/1772368148","heartbeat_front_addr":"172.21.15.169:6829/1772368148","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:31:01.341467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:31:39.201921+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:32:17.444673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:32:56.146463+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:33:30.154703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:34:06.920388+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:34:43.105186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:35:20.969170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6801/3947556342":"2023-12-17T07:35:49.313767+0000","172.21.15.7:6800/3947556342":"2023-12-17T07:35:49.313767+0000","172.21.15.7:0/3815256908":"2023-12-17T07:35:49.313767+0000","172.21.15.7:0/2722441948":"2023-12-17T07:29:15.253163+0000","172.21.15.7:0/2431018109":"2023-12-17T07:29:15.253163+0000","172.21.15.7:6800/847724836":"2023-12-17T07:28:42.956756+0000","172.21.15.7:0/2086785434":"2023-12-17T07:35:49.313767+0000","172.21.15.7:6801/2454016267":"2023-12-17T07:29:15.253163+0000","172.21.15.7:6801/847724836":"2023-12-17T07:28:42.956756+0000","172.21.15.7:0/2442196965":"2023-12-17T07:35:49.313767+0000","172.21.15.7:0/1208557731":"2023-12-17T07:29:15.253163+0000","172.21.15.7:6800/2454016267":"2023-12-17T07:29:15.253163+0000","172.21.15.7:0/2162341476":"2023-12-17T07:28:42.956756+0000","172.21.15.7:0/3673321318":"2023-12-17T07:28:42.956756+0000","172.21.15.7:0/3972277094":"2023-12-17T07:35:49.313767+0000","172.21.15.7:0/3086453482":"2023-12-17T07:28:42.956756+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T07:36:29.594 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-12-16T07:36:29.594 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph osd dump --format=json 2023-12-16T07:36:30.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:29 smithi007 bash[25009]: cluster 2023-12-16T07:36:28.193403+0000 mgr.y (mgr.24367) 39 : cluster [DBG] pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:30.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:29 smithi007 bash[25009]: audit 2023-12-16T07:36:28.853830+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.7:0/3821578093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T07:36:30.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:29 smithi007 bash[30436]: cluster 2023-12-16T07:36:28.193403+0000 mgr.y (mgr.24367) 39 : cluster [DBG] pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:30.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:29 smithi007 bash[30436]: audit 2023-12-16T07:36:28.853830+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.7:0/3821578093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T07:36:30.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:29 smithi169 bash[25083]: cluster 2023-12-16T07:36:28.193403+0000 mgr.y (mgr.24367) 39 : cluster [DBG] pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:30.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:29 smithi169 bash[25083]: audit 2023-12-16T07:36:28.853830+0000 mon.c (mon.1) 12 : audit [DBG] from='client.? 172.21.15.7:0/3821578093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T07:36:32.182 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:31 smithi169 bash[25083]: cluster 2023-12-16T07:36:30.193999+0000 mgr.y (mgr.24367) 40 : cluster [DBG] pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:32.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:31 smithi007 bash[25009]: cluster 2023-12-16T07:36:30.193999+0000 mgr.y (mgr.24367) 40 : cluster [DBG] pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:32.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:31 smithi007 bash[30436]: cluster 2023-12-16T07:36:30.193999+0000 mgr.y (mgr.24367) 40 : cluster [DBG] pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:32.333 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:32.682 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:36:32 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:36:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:36:33.401 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:33 smithi007 bash[30436]: cluster 2023-12-16T07:36:32.194682+0000 mgr.y (mgr.24367) 41 : cluster [DBG] pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:33.402 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:33 smithi007 bash[25009]: cluster 2023-12-16T07:36:32.194682+0000 mgr.y (mgr.24367) 41 : cluster [DBG] pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:33.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:33 smithi169 bash[25083]: cluster 2023-12-16T07:36:32.194682+0000 mgr.y (mgr.24367) 41 : cluster [DBG] pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:34.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:36:33 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:36:33] "GET /metrics HTTP/1.1" 200 191505 "" "Prometheus/2.33.4" 2023-12-16T07:36:35.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:35 smithi007 bash[25009]: cluster 2023-12-16T07:36:34.195276+0000 mgr.y (mgr.24367) 42 : cluster [DBG] pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:35.253 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:35 smithi007 bash[30436]: cluster 2023-12-16T07:36:34.195276+0000 mgr.y (mgr.24367) 42 : cluster [DBG] pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:35.380 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-16T07:36:35.380 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":59,"fsid":"8b71d3dc-9be4-11ee-95a5-87774f69a715","created":"2023-12-16T07:28:10.358509+0000","modified":"2023-12-16T07:35:49.313881+0000","last_up_change":"2023-12-16T07:35:22.069655+0000","last_in_change":"2023-12-16T07:34:57.664166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-16T07:32:19.902470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"d8aa3a96-fde6-48ee-9a0a-4ad9a0d4f427","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":3002763011},{"type":"v1","addr":"172.21.15.7:6803","nonce":3002763011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":3002763011},{"type":"v1","addr":"172.21.15.7:6805","nonce":3002763011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":3002763011},{"type":"v1","addr":"172.21.15.7:6809","nonce":3002763011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":3002763011},{"type":"v1","addr":"172.21.15.7:6807","nonce":3002763011}]},"public_addr":"172.21.15.7:6803/3002763011","cluster_addr":"172.21.15.7:6805/3002763011","heartbeat_back_addr":"172.21.15.7:6809/3002763011","heartbeat_front_addr":"172.21.15.7:6807/3002763011","state":["exists","up"]},{"osd":1,"uuid":"b52565e1-b236-4f11-8356-c6e90f476298","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":4053186404},{"type":"v1","addr":"172.21.15.7:6811","nonce":4053186404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":4053186404},{"type":"v1","addr":"172.21.15.7:6813","nonce":4053186404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":4053186404},{"type":"v1","addr":"172.21.15.7:6817","nonce":4053186404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":4053186404},{"type":"v1","addr":"172.21.15.7:6815","nonce":4053186404}]},"public_addr":"172.21.15.7:6811/4053186404","cluster_addr":"172.21.15.7:6813/4053186404","heartbeat_back_addr":"172.21.15.7:6817/4053186404","heartbeat_front_addr":"172.21.15.7:6815/4053186404","state":["exists","up"]},{"osd":2,"uuid":"5a43c2e0-1837-4880-b904-5ea6041d45a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":2173677438},{"type":"v1","addr":"172.21.15.7:6819","nonce":2173677438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":2173677438},{"type":"v1","addr":"172.21.15.7:6821","nonce":2173677438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":2173677438},{"type":"v1","addr":"172.21.15.7:6825","nonce":2173677438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":2173677438},{"type":"v1","addr":"172.21.15.7:6823","nonce":2173677438}]},"public_addr":"172.21.15.7:6819/2173677438","cluster_addr":"172.21.15.7:6821/2173677438","heartbeat_back_addr":"172.21.15.7:6825/2173677438","heartbeat_front_addr":"172.21.15.7:6823/2173677438","state":["exists","up"]},{"osd":3,"uuid":"4776e261-fa8a-4652-a32f-83710dda5dbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":1623735309},{"type":"v1","addr":"172.21.15.7:6827","nonce":1623735309}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":1623735309},{"type":"v1","addr":"172.21.15.7:6829","nonce":1623735309}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":1623735309},{"type":"v1","addr":"172.21.15.7:6833","nonce":1623735309}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":1623735309},{"type":"v1","addr":"172.21.15.7:6831","nonce":1623735309}]},"public_addr":"172.21.15.7:6827/1623735309","cluster_addr":"172.21.15.7:6829/1623735309","heartbeat_back_addr":"172.21.15.7:6833/1623735309","heartbeat_front_addr":"172.21.15.7:6831/1623735309","state":["exists","up"]},{"osd":4,"uuid":"d5631b50-8179-4ed7-8f03-4de60d19bb69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":3747189467},{"type":"v1","addr":"172.21.15.169:6801","nonce":3747189467}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":3747189467},{"type":"v1","addr":"172.21.15.169:6803","nonce":3747189467}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6806","nonce":3747189467},{"type":"v1","addr":"172.21.15.169:6807","nonce":3747189467}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":3747189467},{"type":"v1","addr":"172.21.15.169:6805","nonce":3747189467}]},"public_addr":"172.21.15.169:6801/3747189467","cluster_addr":"172.21.15.169:6803/3747189467","heartbeat_back_addr":"172.21.15.169:6807/3747189467","heartbeat_front_addr":"172.21.15.169:6805/3747189467","state":["exists","up"]},{"osd":5,"uuid":"70f322c0-b8b5-439c-bf2a-8688a28ca919","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":3877143549},{"type":"v1","addr":"172.21.15.169:6809","nonce":3877143549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":3877143549},{"type":"v1","addr":"172.21.15.169:6811","nonce":3877143549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":3877143549},{"type":"v1","addr":"172.21.15.169:6815","nonce":3877143549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":3877143549},{"type":"v1","addr":"172.21.15.169:6813","nonce":3877143549}]},"public_addr":"172.21.15.169:6809/3877143549","cluster_addr":"172.21.15.169:6811/3877143549","heartbeat_back_addr":"172.21.15.169:6815/3877143549","heartbeat_front_addr":"172.21.15.169:6813/3877143549","state":["exists","up"]},{"osd":6,"uuid":"75186dc4-13a9-4c53-9db6-07724e058c74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6816","nonce":1373876707},{"type":"v1","addr":"172.21.15.169:6817","nonce":1373876707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6818","nonce":1373876707},{"type":"v1","addr":"172.21.15.169:6819","nonce":1373876707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6822","nonce":1373876707},{"type":"v1","addr":"172.21.15.169:6823","nonce":1373876707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6820","nonce":1373876707},{"type":"v1","addr":"172.21.15.169:6821","nonce":1373876707}]},"public_addr":"172.21.15.169:6817/1373876707","cluster_addr":"172.21.15.169:6819/1373876707","heartbeat_back_addr":"172.21.15.169:6823/1373876707","heartbeat_front_addr":"172.21.15.169:6821/1373876707","state":["exists","up"]},{"osd":7,"uuid":"1f760eeb-da8d-4fce-a8f4-cf42d952fcc2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6824","nonce":1772368148},{"type":"v1","addr":"172.21.15.169:6825","nonce":1772368148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6826","nonce":1772368148},{"type":"v1","addr":"172.21.15.169:6827","nonce":1772368148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6830","nonce":1772368148},{"type":"v1","addr":"172.21.15.169:6831","nonce":1772368148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6828","nonce":1772368148},{"type":"v1","addr":"172.21.15.169:6829","nonce":1772368148}]},"public_addr":"172.21.15.169:6825/1772368148","cluster_addr":"172.21.15.169:6827/1772368148","heartbeat_back_addr":"172.21.15.169:6831/1772368148","heartbeat_front_addr":"172.21.15.169:6829/1772368148","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:31:01.341467+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:31:39.201921+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:32:17.444673+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:32:56.146463+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:33:30.154703+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:34:06.920388+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:34:43.105186+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2023-12-16T07:35:20.969170+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:6801/3947556342":"2023-12-17T07:35:49.313767+0000","172.21.15.7:6800/3947556342":"2023-12-17T07:35:49.313767+0000","172.21.15.7:0/3815256908":"2023-12-17T07:35:49.313767+0000","172.21.15.7:0/2722441948":"2023-12-17T07:29:15.253163+0000","172.21.15.7:0/2431018109":"2023-12-17T07:29:15.253163+0000","172.21.15.7:6800/847724836":"2023-12-17T07:28:42.956756+0000","172.21.15.7:0/2086785434":"2023-12-17T07:35:49.313767+0000","172.21.15.7:6801/2454016267":"2023-12-17T07:29:15.253163+0000","172.21.15.7:6801/847724836":"2023-12-17T07:28:42.956756+0000","172.21.15.7:0/2442196965":"2023-12-17T07:35:49.313767+0000","172.21.15.7:0/1208557731":"2023-12-17T07:29:15.253163+0000","172.21.15.7:6800/2454016267":"2023-12-17T07:29:15.253163+0000","172.21.15.7:0/2162341476":"2023-12-17T07:28:42.956756+0000","172.21.15.7:0/3673321318":"2023-12-17T07:28:42.956756+0000","172.21.15.7:0/3972277094":"2023-12-17T07:35:49.313767+0000","172.21.15.7:0/3086453482":"2023-12-17T07:28:42.956756+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-16T07:36:35.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:35 smithi169 bash[25083]: cluster 2023-12-16T07:36:34.195276+0000 mgr.y (mgr.24367) 42 : cluster [DBG] pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:36.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2023-12-16T07:36:36.182 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2023-12-16T07:36:36.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2023-12-16T07:36:36.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2023-12-16T07:36:36.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2023-12-16T07:36:36.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2023-12-16T07:36:36.184 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2023-12-16T07:36:36.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2023-12-16T07:36:36.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:36 smithi007 bash[25009]: audit 2023-12-16T07:36:35.377711+0000 mon.a (mon.0) 653 : audit [DBG] from='client.? 172.21.15.7:0/3204966271' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T07:36:36.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:36 smithi007 bash[30436]: audit 2023-12-16T07:36:35.377711+0000 mon.a (mon.0) 653 : audit [DBG] from='client.? 172.21.15.7:0/3204966271' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T07:36:36.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:36 smithi169 bash[25083]: audit 2023-12-16T07:36:35.377711+0000 mon.a (mon.0) 653 : audit [DBG] from='client.? 172.21.15.7:0/3204966271' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-12-16T07:36:37.066 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:36:36 smithi007 bash[45990]: level=info ts=2023-12-16T07:36:36.658Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.00489124s 2023-12-16T07:36:37.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:37 smithi007 bash[25009]: cluster 2023-12-16T07:36:36.196502+0000 mgr.y (mgr.24367) 43 : cluster [DBG] pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:37.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:37 smithi007 bash[30436]: cluster 2023-12-16T07:36:36.196502+0000 mgr.y (mgr.24367) 43 : cluster [DBG] pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:37.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:37 smithi169 bash[25083]: cluster 2023-12-16T07:36:36.196502+0000 mgr.y (mgr.24367) 43 : cluster [DBG] pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:39.024 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:39.024 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:39.026 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:39.026 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:39.026 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:39.027 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:39.027 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:39.028 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:39.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:39 smithi007 bash[25009]: cluster 2023-12-16T07:36:38.197106+0000 mgr.y (mgr.24367) 44 : cluster [DBG] pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:39.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:39 smithi007 bash[30436]: cluster 2023-12-16T07:36:38.197106+0000 mgr.y (mgr.24367) 44 : cluster [DBG] pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:39.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:39 smithi169 bash[25083]: cluster 2023-12-16T07:36:38.197106+0000 mgr.y (mgr.24367) 44 : cluster [DBG] pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:41.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:41 smithi007 bash[25009]: cluster 2023-12-16T07:36:40.197709+0000 mgr.y (mgr.24367) 45 : cluster [DBG] pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:41.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:41 smithi007 bash[30436]: cluster 2023-12-16T07:36:40.197709+0000 mgr.y (mgr.24367) 45 : cluster [DBG] pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:41.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:41 smithi169 bash[25083]: cluster 2023-12-16T07:36:40.197709+0000 mgr.y (mgr.24367) 45 : cluster [DBG] pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:42.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:36:42 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:36:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:36:43.432 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:43 smithi169 bash[25083]: cluster 2023-12-16T07:36:42.198372+0000 mgr.y (mgr.24367) 46 : cluster [DBG] pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:44.118 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:43 smithi007 bash[30436]: cluster 2023-12-16T07:36:42.198372+0000 mgr.y (mgr.24367) 46 : cluster [DBG] pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:44.119 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:43 smithi007 bash[25009]: cluster 2023-12-16T07:36:42.198372+0000 mgr.y (mgr.24367) 46 : cluster [DBG] pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:44.119 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:36:43 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:36:43] "GET /metrics HTTP/1.1" 200 191493 "" "Prometheus/2.33.4" 2023-12-16T07:36:45.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:45 smithi007 bash[30436]: cluster 2023-12-16T07:36:44.199026+0000 mgr.y (mgr.24367) 47 : cluster [DBG] pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:45.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:45 smithi007 bash[25009]: cluster 2023-12-16T07:36:44.199026+0000 mgr.y (mgr.24367) 47 : cluster [DBG] pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:45.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:45 smithi169 bash[25083]: cluster 2023-12-16T07:36:44.199026+0000 mgr.y (mgr.24367) 47 : cluster [DBG] pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:47.426 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[25083]: audit 2023-12-16T07:36:46.153874+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:47.426 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[25083]: audit 2023-12-16T07:36:46.156947+0000 mon.a (mon.0) 655 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:36:47.426 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[25083]: audit 2023-12-16T07:36:46.159280+0000 mon.a (mon.0) 656 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:36:47.426 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[25083]: audit 2023-12-16T07:36:46.160722+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:36:47.426 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[25083]: cluster 2023-12-16T07:36:46.199607+0000 mgr.y (mgr.24367) 48 : cluster [DBG] pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:47.426 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2023-12-16T07:36:47.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:47 smithi007 bash[30436]: audit 2023-12-16T07:36:46.153874+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:47.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:47 smithi007 bash[30436]: audit 2023-12-16T07:36:46.156947+0000 mon.a (mon.0) 655 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:36:47.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:47 smithi007 bash[30436]: audit 2023-12-16T07:36:46.159280+0000 mon.a (mon.0) 656 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:36:47.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:47 smithi007 bash[30436]: audit 2023-12-16T07:36:46.160722+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:36:47.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:47 smithi007 bash[30436]: cluster 2023-12-16T07:36:46.199607+0000 mgr.y (mgr.24367) 48 : cluster [DBG] pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:47.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:47 smithi007 bash[25009]: audit 2023-12-16T07:36:46.153874+0000 mon.a (mon.0) 654 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:47.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:47 smithi007 bash[25009]: audit 2023-12-16T07:36:46.156947+0000 mon.a (mon.0) 655 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:36:47.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:47 smithi007 bash[25009]: audit 2023-12-16T07:36:46.159280+0000 mon.a (mon.0) 656 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:36:47.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:47 smithi007 bash[25009]: audit 2023-12-16T07:36:46.160722+0000 mon.a (mon.0) 657 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:36:47.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:47 smithi007 bash[25009]: cluster 2023-12-16T07:36:46.199607+0000 mgr.y (mgr.24367) 48 : cluster [DBG] pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:47.683 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2023-12-16T07:36:47.683 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2023-12-16T07:36:47.683 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2023-12-16T07:36:47.683 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2023-12-16T07:36:47.683 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2023-12-16T07:36:47.683 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2023-12-16T07:36:47.683 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2023-12-16T07:36:47.683 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2023-12-16T07:36:47.684 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2023-12-16T07:36:47.973 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2023-12-16T07:36:47.973 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2023-12-16T07:36:47.973 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2023-12-16T07:36:47.973 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2023-12-16T07:36:47.973 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2023-12-16T07:36:47.973 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2023-12-16T07:36:47.973 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2023-12-16T07:36:47.973 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2023-12-16T07:36:47.973 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:47 smithi169 bash[40964]: t=2023-12-16T07:36:47+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2023-12-16T07:36:48.265 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2023-12-16T07:36:48.266 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2023-12-16T07:36:48.266 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2023-12-16T07:36:48.266 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2023-12-16T07:36:48.266 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2023-12-16T07:36:48.267 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2023-12-16T07:36:48.267 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2023-12-16T07:36:48.267 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2023-12-16T07:36:48.529 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2023-12-16T07:36:48.529 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2023-12-16T07:36:48.529 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2023-12-16T07:36:48.529 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2023-12-16T07:36:48.529 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2023-12-16T07:36:48.529 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2023-12-16T07:36:48.529 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2023-12-16T07:36:48.529 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2023-12-16T07:36:48.798 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2023-12-16T07:36:48.798 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2023-12-16T07:36:48.798 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2023-12-16T07:36:48.798 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2023-12-16T07:36:48.798 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2023-12-16T07:36:48.798 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2023-12-16T07:36:48.798 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2023-12-16T07:36:49.059 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2023-12-16T07:36:49.059 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2023-12-16T07:36:49.060 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2023-12-16T07:36:49.060 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2023-12-16T07:36:49.060 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2023-12-16T07:36:49.060 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2023-12-16T07:36:49.060 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:48 smithi169 bash[40964]: t=2023-12-16T07:36:48+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2023-12-16T07:36:49.060 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2023-12-16T07:36:49.060 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2023-12-16T07:36:49.311 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[25083]: cluster 2023-12-16T07:36:48.200298+0000 mgr.y (mgr.24367) 49 : cluster [DBG] pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:49.311 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2023-12-16T07:36:49.312 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2023-12-16T07:36:49.312 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2023-12-16T07:36:49.312 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2023-12-16T07:36:49.312 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2023-12-16T07:36:49.312 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2023-12-16T07:36:49.312 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2023-12-16T07:36:49.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:49 smithi007 bash[30436]: cluster 2023-12-16T07:36:48.200298+0000 mgr.y (mgr.24367) 49 : cluster [DBG] pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:49.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:49 smithi007 bash[25009]: cluster 2023-12-16T07:36:48.200298+0000 mgr.y (mgr.24367) 49 : cluster [DBG] pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:49.573 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2023-12-16T07:36:49.573 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2023-12-16T07:36:49.573 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2023-12-16T07:36:49.573 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2023-12-16T07:36:49.573 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2023-12-16T07:36:49.573 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2023-12-16T07:36:49.573 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2023-12-16T07:36:49.573 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2023-12-16T07:36:49.573 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2023-12-16T07:36:49.842 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2023-12-16T07:36:49.842 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2023-12-16T07:36:49.842 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2023-12-16T07:36:49.843 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2023-12-16T07:36:49.843 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2023-12-16T07:36:49.843 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2023-12-16T07:36:49.843 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2023-12-16T07:36:49.843 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2023-12-16T07:36:50.112 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2023-12-16T07:36:50.112 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2023-12-16T07:36:50.112 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2023-12-16T07:36:50.112 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:49 smithi169 bash[40964]: t=2023-12-16T07:36:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2023-12-16T07:36:50.112 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2023-12-16T07:36:50.112 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2023-12-16T07:36:50.112 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2023-12-16T07:36:50.322 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2023-12-16T07:36:50.322 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2023-12-16T07:36:50.323 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2023-12-16T07:36:50.323 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2023-12-16T07:36:50.323 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2023-12-16T07:36:50.323 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2023-12-16T07:36:50.398 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2023-12-16T07:36:50.683 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2023-12-16T07:36:50.683 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2023-12-16T07:36:50.683 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2023-12-16T07:36:50.683 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2023-12-16T07:36:50.683 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2023-12-16T07:36:50.683 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2023-12-16T07:36:50.766 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2023-12-16T07:36:50.945 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2023-12-16T07:36:50.945 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2023-12-16T07:36:50.945 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2023-12-16T07:36:50.945 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2023-12-16T07:36:51.231 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2023-12-16T07:36:51.232 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:50 smithi169 bash[40964]: t=2023-12-16T07:36:50+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2023-12-16T07:36:51.232 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2023-12-16T07:36:51.232 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2023-12-16T07:36:51.232 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2023-12-16T07:36:51.232 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2023-12-16T07:36:51.232 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2023-12-16T07:36:51.232 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2023-12-16T07:36:51.232 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2023-12-16T07:36:51.500 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[25083]: cluster 2023-12-16T07:36:50.200918+0000 mgr.y (mgr.24367) 50 : cluster [DBG] pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:51.500 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[25083]: audit 2023-12-16T07:36:50.297722+0000 mon.a (mon.0) 658 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:51.501 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[25083]: audit 2023-12-16T07:36:50.523638+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:36:51.501 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[25083]: audit 2023-12-16T07:36:50.533583+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:36:51.501 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[25083]: audit 2023-12-16T07:36:50.959744+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:51.501 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2023-12-16T07:36:51.501 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2023-12-16T07:36:51.501 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2023-12-16T07:36:51.501 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2023-12-16T07:36:51.501 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2023-12-16T07:36:51.502 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2023-12-16T07:36:51.576 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:51 smithi007 bash[30436]: cluster 2023-12-16T07:36:50.200918+0000 mgr.y (mgr.24367) 50 : cluster [DBG] pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:51.576 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:51 smithi007 bash[30436]: audit 2023-12-16T07:36:50.297722+0000 mon.a (mon.0) 658 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:51.576 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:51 smithi007 bash[30436]: audit 2023-12-16T07:36:50.523638+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:36:51.576 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:51 smithi007 bash[30436]: audit 2023-12-16T07:36:50.533583+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:36:51.576 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:51 smithi007 bash[30436]: audit 2023-12-16T07:36:50.959744+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:51.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:51 smithi007 bash[25009]: cluster 2023-12-16T07:36:50.200918+0000 mgr.y (mgr.24367) 50 : cluster [DBG] pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:51.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:51 smithi007 bash[25009]: audit 2023-12-16T07:36:50.297722+0000 mon.a (mon.0) 658 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:51.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:51 smithi007 bash[25009]: audit 2023-12-16T07:36:50.523638+0000 mon.a (mon.0) 659 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:36:51.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:51 smithi007 bash[25009]: audit 2023-12-16T07:36:50.533583+0000 mon.a (mon.0) 660 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:36:51.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:51 smithi007 bash[25009]: audit 2023-12-16T07:36:50.959744+0000 mon.a (mon.0) 661 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:51.753 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2023-12-16T07:36:51.754 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2023-12-16T07:36:51.754 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2023-12-16T07:36:51.754 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2023-12-16T07:36:51.754 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2023-12-16T07:36:51.754 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2023-12-16T07:36:51.754 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2023-12-16T07:36:51.754 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2023-12-16T07:36:51.754 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2023-12-16T07:36:52.006 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2023-12-16T07:36:52.006 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2023-12-16T07:36:52.006 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2023-12-16T07:36:52.006 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2023-12-16T07:36:52.006 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2023-12-16T07:36:52.006 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2023-12-16T07:36:52.006 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:51 smithi169 bash[40964]: t=2023-12-16T07:36:51+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2023-12-16T07:36:52.006 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2023-12-16T07:36:52.246 INFO:teuthology.orchestra.run.smithi007.stdout:197568495642 2023-12-16T07:36:52.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.6 2023-12-16T07:36:52.258 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:36:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:36:52.259 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2023-12-16T07:36:52.259 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2023-12-16T07:36:52.259 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2023-12-16T07:36:52.259 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2023-12-16T07:36:52.259 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2023-12-16T07:36:52.259 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2023-12-16T07:36:52.259 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2023-12-16T07:36:52.259 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2023-12-16T07:36:52.536 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2023-12-16T07:36:52.536 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2023-12-16T07:36:52.536 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2023-12-16T07:36:52.536 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2023-12-16T07:36:52.536 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2023-12-16T07:36:52.536 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2023-12-16T07:36:52.536 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2023-12-16T07:36:52.602 INFO:teuthology.orchestra.run.smithi007.stdout:55834574910 2023-12-16T07:36:52.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.1 2023-12-16T07:36:52.789 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2023-12-16T07:36:52.789 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2023-12-16T07:36:52.789 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2023-12-16T07:36:52.789 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2023-12-16T07:36:52.789 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2023-12-16T07:36:52.790 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2023-12-16T07:36:52.790 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2023-12-16T07:36:52.790 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2023-12-16T07:36:53.042 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2023-12-16T07:36:53.042 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2023-12-16T07:36:53.042 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2023-12-16T07:36:53.042 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2023-12-16T07:36:53.043 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2023-12-16T07:36:53.043 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2023-12-16T07:36:53.043 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2023-12-16T07:36:53.043 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:52 smithi169 bash[40964]: t=2023-12-16T07:36:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2023-12-16T07:36:53.043 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2023-12-16T07:36:53.310 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[25083]: cluster 2023-12-16T07:36:52.201669+0000 mgr.y (mgr.24367) 51 : cluster [DBG] pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:53.311 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2023-12-16T07:36:53.311 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2023-12-16T07:36:53.311 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2023-12-16T07:36:53.311 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2023-12-16T07:36:53.311 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2023-12-16T07:36:53.311 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2023-12-16T07:36:53.552 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:53 smithi007 bash[30436]: cluster 2023-12-16T07:36:52.201669+0000 mgr.y (mgr.24367) 51 : cluster [DBG] pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:53.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:53 smithi007 bash[25009]: cluster 2023-12-16T07:36:52.201669+0000 mgr.y (mgr.24367) 51 : cluster [DBG] pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:53.572 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2023-12-16T07:36:53.572 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2023-12-16T07:36:53.572 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2023-12-16T07:36:53.572 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2023-12-16T07:36:53.572 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2023-12-16T07:36:53.572 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2023-12-16T07:36:53.572 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2023-12-16T07:36:53.572 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2023-12-16T07:36:53.816 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:36:53 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:36:53] "GET /metrics HTTP/1.1" 200 191499 "" "Prometheus/2.33.4" 2023-12-16T07:36:53.832 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2023-12-16T07:36:53.832 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2023-12-16T07:36:53.833 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2023-12-16T07:36:53.833 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2023-12-16T07:36:53.833 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2023-12-16T07:36:53.833 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2023-12-16T07:36:53.833 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2023-12-16T07:36:54.152 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2023-12-16T07:36:54.152 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2023-12-16T07:36:54.152 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2023-12-16T07:36:54.152 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2023-12-16T07:36:54.152 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2023-12-16T07:36:54.152 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:53 smithi169 bash[40964]: t=2023-12-16T07:36:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2023-12-16T07:36:54.152 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2023-12-16T07:36:54.152 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2023-12-16T07:36:54.152 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2023-12-16T07:36:54.413 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2023-12-16T07:36:54.413 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2023-12-16T07:36:54.413 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2023-12-16T07:36:54.413 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2023-12-16T07:36:54.414 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2023-12-16T07:36:54.414 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2023-12-16T07:36:54.414 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2023-12-16T07:36:54.665 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2023-12-16T07:36:54.666 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2023-12-16T07:36:54.666 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2023-12-16T07:36:54.666 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2023-12-16T07:36:54.666 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2023-12-16T07:36:54.666 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2023-12-16T07:36:54.802 INFO:teuthology.orchestra.run.smithi007.stdout:128849018921 2023-12-16T07:36:54.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.4 2023-12-16T07:36:54.933 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2023-12-16T07:36:54.933 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2023-12-16T07:36:54.933 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2023-12-16T07:36:54.933 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2023-12-16T07:36:54.933 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2023-12-16T07:36:54.933 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2023-12-16T07:36:54.933 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2023-12-16T07:36:54.934 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2023-12-16T07:36:54.934 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2023-12-16T07:36:55.212 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:54 smithi169 bash[40964]: t=2023-12-16T07:36:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2023-12-16T07:36:55.212 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2023-12-16T07:36:55.212 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2023-12-16T07:36:55.213 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2023-12-16T07:36:55.213 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2023-12-16T07:36:55.213 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2023-12-16T07:36:55.213 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2023-12-16T07:36:55.213 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2023-12-16T07:36:55.482 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[25083]: cluster 2023-12-16T07:36:54.202312+0000 mgr.y (mgr.24367) 52 : cluster [DBG] pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:55.482 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[25083]: audit 2023-12-16T07:36:54.437810+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:55.482 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[25083]: audit 2023-12-16T07:36:54.707466+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:55.482 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[25083]: cephadm 2023-12-16T07:36:54.712633+0000 mgr.y (mgr.24367) 53 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-12-16T07:36:55.482 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[25083]: cephadm 2023-12-16T07:36:54.716859+0000 mgr.y (mgr.24367) 54 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi007 2023-12-16T07:36:55.483 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2023-12-16T07:36:55.483 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2023-12-16T07:36:55.483 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2023-12-16T07:36:55.483 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2023-12-16T07:36:55.483 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2023-12-16T07:36:55.483 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2023-12-16T07:36:55.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:55 smithi007 bash[25009]: cluster 2023-12-16T07:36:54.202312+0000 mgr.y (mgr.24367) 52 : cluster [DBG] pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:55.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:55 smithi007 bash[25009]: audit 2023-12-16T07:36:54.437810+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:55.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:55 smithi007 bash[25009]: audit 2023-12-16T07:36:54.707466+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:55.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:55 smithi007 bash[25009]: cephadm 2023-12-16T07:36:54.712633+0000 mgr.y (mgr.24367) 53 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-12-16T07:36:55.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:55 smithi007 bash[25009]: cephadm 2023-12-16T07:36:54.716859+0000 mgr.y (mgr.24367) 54 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi007 2023-12-16T07:36:55.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:55 smithi007 bash[30436]: cluster 2023-12-16T07:36:54.202312+0000 mgr.y (mgr.24367) 52 : cluster [DBG] pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:55.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:55 smithi007 bash[30436]: audit 2023-12-16T07:36:54.437810+0000 mon.a (mon.0) 662 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:55.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:55 smithi007 bash[30436]: audit 2023-12-16T07:36:54.707466+0000 mon.a (mon.0) 663 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:36:55.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:55 smithi007 bash[30436]: cephadm 2023-12-16T07:36:54.712633+0000 mgr.y (mgr.24367) 53 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-12-16T07:36:55.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:55 smithi007 bash[30436]: cephadm 2023-12-16T07:36:54.716859+0000 mgr.y (mgr.24367) 54 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi007 2023-12-16T07:36:55.734 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2023-12-16T07:36:55.734 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2023-12-16T07:36:55.734 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2023-12-16T07:36:55.735 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2023-12-16T07:36:55.735 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2023-12-16T07:36:55.735 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2023-12-16T07:36:55.735 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2023-12-16T07:36:55.735 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2023-12-16T07:36:55.735 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2023-12-16T07:36:55.995 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2023-12-16T07:36:55.995 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2023-12-16T07:36:55.995 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2023-12-16T07:36:55.995 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2023-12-16T07:36:55.995 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2023-12-16T07:36:55.995 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2023-12-16T07:36:55.996 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2023-12-16T07:36:55.996 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:55 smithi169 bash[40964]: t=2023-12-16T07:36:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2023-12-16T07:36:56.265 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2023-12-16T07:36:56.265 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2023-12-16T07:36:56.265 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2023-12-16T07:36:56.266 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2023-12-16T07:36:56.266 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2023-12-16T07:36:56.266 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2023-12-16T07:36:56.266 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2023-12-16T07:36:56.266 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2023-12-16T07:36:56.518 INFO:teuthology.orchestra.run.smithi007.stdout:34359738439 2023-12-16T07:36:56.518 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.0 2023-12-16T07:36:56.519 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2023-12-16T07:36:56.519 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2023-12-16T07:36:56.519 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2023-12-16T07:36:56.519 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2023-12-16T07:36:56.519 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2023-12-16T07:36:56.519 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2023-12-16T07:36:56.519 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2023-12-16T07:36:56.519 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2023-12-16T07:36:56.828 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2023-12-16T07:36:56.829 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2023-12-16T07:36:56.829 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2023-12-16T07:36:56.829 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2023-12-16T07:36:56.829 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2023-12-16T07:36:56.829 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2023-12-16T07:36:56.829 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2023-12-16T07:36:56.829 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2023-12-16T07:36:56.829 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2023-12-16T07:36:56.893 INFO:teuthology.orchestra.run.smithi007.stdout:231928234003 2023-12-16T07:36:56.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.7 2023-12-16T07:36:56.902 INFO:teuthology.orchestra.run.smithi007.stdout:163208757282 2023-12-16T07:36:56.903 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.5 2023-12-16T07:36:56.903 INFO:teuthology.orchestra.run.smithi007.stdout:107374182447 2023-12-16T07:36:56.904 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.3 2023-12-16T07:36:57.082 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2023-12-16T07:36:57.082 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2023-12-16T07:36:57.082 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2023-12-16T07:36:57.082 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2023-12-16T07:36:57.082 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:56 smithi169 bash[40964]: t=2023-12-16T07:36:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2023-12-16T07:36:57.082 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2023-12-16T07:36:57.082 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2023-12-16T07:36:57.082 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2023-12-16T07:36:57.350 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2023-12-16T07:36:57.351 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2023-12-16T07:36:57.351 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2023-12-16T07:36:57.351 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2023-12-16T07:36:57.351 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2023-12-16T07:36:57.351 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2023-12-16T07:36:57.351 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2023-12-16T07:36:57.603 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[25083]: cluster 2023-12-16T07:36:56.202985+0000 mgr.y (mgr.24367) 55 : cluster [DBG] pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:57.603 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2023-12-16T07:36:57.603 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2023-12-16T07:36:57.603 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2023-12-16T07:36:57.603 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2023-12-16T07:36:57.603 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2023-12-16T07:36:57.603 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2023-12-16T07:36:57.604 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2023-12-16T07:36:57.604 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2023-12-16T07:36:57.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:57 smithi007 bash[30436]: cluster 2023-12-16T07:36:56.202985+0000 mgr.y (mgr.24367) 55 : cluster [DBG] pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:57.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:57 smithi007 bash[25009]: cluster 2023-12-16T07:36:56.202985+0000 mgr.y (mgr.24367) 55 : cluster [DBG] pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:57.856 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2023-12-16T07:36:57.856 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2023-12-16T07:36:57.856 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2023-12-16T07:36:57.856 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2023-12-16T07:36:57.856 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2023-12-16T07:36:57.856 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2023-12-16T07:36:57.857 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2023-12-16T07:36:58.100 INFO:teuthology.orchestra.run.smithi007.stdout:77309411384 2023-12-16T07:36:58.100 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph osd last-stat-seq osd.2 2023-12-16T07:36:58.108 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2023-12-16T07:36:58.108 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2023-12-16T07:36:58.108 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:57 smithi169 bash[40964]: t=2023-12-16T07:36:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2023-12-16T07:36:58.108 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2023-12-16T07:36:58.108 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2023-12-16T07:36:58.108 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2023-12-16T07:36:58.108 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2023-12-16T07:36:58.108 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2023-12-16T07:36:58.359 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2023-12-16T07:36:58.359 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2023-12-16T07:36:58.360 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2023-12-16T07:36:58.360 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2023-12-16T07:36:58.360 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2023-12-16T07:36:58.360 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2023-12-16T07:36:58.480 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:58.481 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:58.629 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2023-12-16T07:36:58.629 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2023-12-16T07:36:58.629 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2023-12-16T07:36:58.629 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2023-12-16T07:36:58.629 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2023-12-16T07:36:58.629 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2023-12-16T07:36:58.629 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2023-12-16T07:36:58.629 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2023-12-16T07:36:58.898 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2023-12-16T07:36:58.898 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2023-12-16T07:36:58.898 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2023-12-16T07:36:58.898 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2023-12-16T07:36:58.899 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2023-12-16T07:36:58.899 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2023-12-16T07:36:58.899 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2023-12-16T07:36:58.899 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2023-12-16T07:36:58.899 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2023-12-16T07:36:59.176 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2023-12-16T07:36:59.176 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2023-12-16T07:36:59.176 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:58 smithi169 bash[40964]: t=2023-12-16T07:36:58+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2023-12-16T07:36:59.176 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2023-12-16T07:36:59.176 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2023-12-16T07:36:59.176 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2023-12-16T07:36:59.176 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2023-12-16T07:36:59.176 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2023-12-16T07:36:59.433 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2023-12-16T07:36:59.433 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2023-12-16T07:36:59.433 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2023-12-16T07:36:59.433 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2023-12-16T07:36:59.433 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2023-12-16T07:36:59.433 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2023-12-16T07:36:59.434 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2023-12-16T07:36:59.434 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2023-12-16T07:36:59.434 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2023-12-16T07:36:59.579 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:59.580 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:59.658 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:59.658 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:59.659 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:36:59.663 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:36:59 smithi007 bash[25009]: cluster 2023-12-16T07:36:58.203525+0000 mgr.y (mgr.24367) 56 : cluster [DBG] pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:59.663 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:36:59 smithi007 bash[30436]: cluster 2023-12-16T07:36:58.203525+0000 mgr.y (mgr.24367) 56 : cluster [DBG] pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:59.697 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[25083]: cluster 2023-12-16T07:36:58.203525+0000 mgr.y (mgr.24367) 56 : cluster [DBG] pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:36:59.697 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2023-12-16T07:36:59.697 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2023-12-16T07:36:59.697 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2023-12-16T07:36:59.698 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2023-12-16T07:36:59.698 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2023-12-16T07:36:59.698 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2023-12-16T07:36:59.698 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2023-12-16T07:36:59.958 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2023-12-16T07:36:59.958 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2023-12-16T07:36:59.959 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2023-12-16T07:36:59.959 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2023-12-16T07:36:59.959 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2023-12-16T07:36:59.959 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2023-12-16T07:36:59.959 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2023-12-16T07:36:59.959 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2023-12-16T07:37:00.210 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:36:59 smithi169 bash[40964]: t=2023-12-16T07:36:59+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2023-12-16T07:37:00.210 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2023-12-16T07:37:00.211 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2023-12-16T07:37:00.211 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2023-12-16T07:37:00.211 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2023-12-16T07:37:00.211 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2023-12-16T07:37:00.211 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2023-12-16T07:37:00.211 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2023-12-16T07:37:00.488 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2023-12-16T07:37:00.488 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2023-12-16T07:37:00.488 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2023-12-16T07:37:00.488 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2023-12-16T07:37:00.488 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2023-12-16T07:37:00.488 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2023-12-16T07:37:00.488 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2023-12-16T07:37:00.601 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:37:00.768 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2023-12-16T07:37:00.768 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=13.191669664s 2023-12-16T07:37:00.768 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Created default organization" logger=sqlstore 2023-12-16T07:37:00.768 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2023-12-16T07:37:00.768 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2023-12-16T07:37:00.768 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2023-12-16T07:37:00.769 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2023-12-16T07:37:00.769 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2023-12-16T07:37:00.769 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2023-12-16T07:37:01.183 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2023-12-16T07:37:01.183 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="warming cache for startup" logger=ngalert 2023-12-16T07:37:01.183 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2023-12-16T07:37:01.183 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:37:00 smithi169 bash[40964]: t=2023-12-16T07:37:00+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2023-12-16T07:37:01.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:01 smithi007 bash[30436]: cluster 2023-12-16T07:37:00.204059+0000 mgr.y (mgr.24367) 57 : cluster [DBG] pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:01.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:01 smithi007 bash[25009]: cluster 2023-12-16T07:37:00.204059+0000 mgr.y (mgr.24367) 57 : cluster [DBG] pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:01.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:01 smithi169 bash[25083]: cluster 2023-12-16T07:37:00.204059+0000 mgr.y (mgr.24367) 57 : cluster [DBG] pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:02.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:37:02 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:37:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:37:03.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:03 smithi007 bash[30436]: cluster 2023-12-16T07:37:02.204650+0000 mgr.y (mgr.24367) 58 : cluster [DBG] pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:03.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:03 smithi007 bash[25009]: cluster 2023-12-16T07:37:02.204650+0000 mgr.y (mgr.24367) 58 : cluster [DBG] pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:03.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:03 smithi169 bash[25083]: cluster 2023-12-16T07:37:02.204650+0000 mgr.y (mgr.24367) 58 : cluster [DBG] pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:04.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:37:03 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:37:03] "GET /metrics HTTP/1.1" 200 191499 "" "Prometheus/2.33.4" 2023-12-16T07:37:05.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:05 smithi007 bash[25009]: cluster 2023-12-16T07:37:04.205277+0000 mgr.y (mgr.24367) 59 : cluster [DBG] pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:05.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:05 smithi007 bash[30436]: cluster 2023-12-16T07:37:04.205277+0000 mgr.y (mgr.24367) 59 : cluster [DBG] pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:05.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:05 smithi169 bash[25083]: cluster 2023-12-16T07:37:04.205277+0000 mgr.y (mgr.24367) 59 : cluster [DBG] pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:06.003 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:05 smithi007 systemd[1]: Stopping Ceph alertmanager.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:37:06.004 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:05 smithi007 bash[48926]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-alertmanager.a 2023-12-16T07:37:06.150 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:06 smithi007 bash[45990]: level=info ts=2023-12-16T07:37:05.989Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2023-12-16T07:37:07.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:07 smithi007 bash[30436]: cluster 2023-12-16T07:37:06.205797+0000 mgr.y (mgr.24367) 60 : cluster [DBG] pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:07.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:07 smithi007 bash[25009]: cluster 2023-12-16T07:37:06.205797+0000 mgr.y (mgr.24367) 60 : cluster [DBG] pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:07.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:07 smithi169 bash[25083]: cluster 2023-12-16T07:37:06.205797+0000 mgr.y (mgr.24367) 60 : cluster [DBG] pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:08.538 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:08 smithi007 bash[48935]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-alertmanager-a 2023-12-16T07:37:08.538 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:08 smithi007 bash[49177]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-alertmanager.a 2023-12-16T07:37:09.101 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:09 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@alertmanager.a.service: Succeeded. 2023-12-16T07:37:09.101 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:09 smithi007 systemd[1]: Stopped Ceph alertmanager.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:37:09.101 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:09 smithi007 systemd[1]: Started Ceph alertmanager.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:37:09.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:09 smithi169 bash[25083]: cluster 2023-12-16T07:37:08.206478+0000 mgr.y (mgr.24367) 61 : cluster [DBG] pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:09.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:09 smithi169 bash[25083]: audit 2023-12-16T07:37:09.111646+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:09.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:09 smithi007 bash[25009]: cluster 2023-12-16T07:37:08.206478+0000 mgr.y (mgr.24367) 61 : cluster [DBG] pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:09.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:09 smithi007 bash[25009]: audit 2023-12-16T07:37:09.111646+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:09.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:09 smithi007 bash[30436]: cluster 2023-12-16T07:37:08.206478+0000 mgr.y (mgr.24367) 61 : cluster [DBG] pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:09.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:09 smithi007 bash[30436]: audit 2023-12-16T07:37:09.111646+0000 mon.a (mon.0) 664 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:10.281 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:10 smithi007 bash[30436]: cephadm 2023-12-16T07:37:09.114470+0000 mgr.y (mgr.24367) 62 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-12-16T07:37:10.282 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:10 smithi007 bash[30436]: cephadm 2023-12-16T07:37:09.121632+0000 mgr.y (mgr.24367) 63 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi169 2023-12-16T07:37:10.568 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:10 smithi169 bash[25083]: cephadm 2023-12-16T07:37:09.114470+0000 mgr.y (mgr.24367) 62 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-12-16T07:37:10.568 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:10 smithi169 bash[25083]: cephadm 2023-12-16T07:37:09.121632+0000 mgr.y (mgr.24367) 63 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi169 2023-12-16T07:37:10.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:10 smithi007 bash[25009]: cephadm 2023-12-16T07:37:09.114470+0000 mgr.y (mgr.24367) 62 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-12-16T07:37:10.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:10 smithi007 bash[25009]: cephadm 2023-12-16T07:37:09.121632+0000 mgr.y (mgr.24367) 63 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi169 2023-12-16T07:37:11.410 INFO:teuthology.orchestra.run.smithi007.stdout:55834574914 2023-12-16T07:37:11.411 INFO:teuthology.orchestra.run.smithi007.stdout:197568495646 2023-12-16T07:37:11.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:11 smithi169 bash[25083]: cluster 2023-12-16T07:37:10.207142+0000 mgr.y (mgr.24367) 64 : cluster [DBG] pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:11.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:11 smithi007 bash[30436]: cluster 2023-12-16T07:37:10.207142+0000 mgr.y (mgr.24367) 64 : cluster [DBG] pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:11.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:11 smithi007 bash[25009]: cluster 2023-12-16T07:37:10.207142+0000 mgr.y (mgr.24367) 64 : cluster [DBG] pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:12.403 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:12 smithi169 bash[25083]: audit 2023-12-16T07:37:11.409040+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.7:0/1096695527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-12-16T07:37:12.403 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:12 smithi169 bash[25083]: audit 2023-12-16T07:37:11.410891+0000 mon.a (mon.0) 665 : audit [DBG] from='client.? 172.21.15.7:0/4271062161' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-12-16T07:37:12.404 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:37:12 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:37:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:37:12.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:12 smithi007 bash[25009]: audit 2023-12-16T07:37:11.409040+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.7:0/1096695527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-12-16T07:37:12.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:12 smithi007 bash[25009]: audit 2023-12-16T07:37:11.410891+0000 mon.a (mon.0) 665 : audit [DBG] from='client.? 172.21.15.7:0/4271062161' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-12-16T07:37:12.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:12 smithi007 bash[30436]: audit 2023-12-16T07:37:11.409040+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.7:0/1096695527' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-12-16T07:37:12.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:12 smithi007 bash[30436]: audit 2023-12-16T07:37:11.410891+0000 mon.a (mon.0) 665 : audit [DBG] from='client.? 172.21.15.7:0/4271062161' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-12-16T07:37:13.183 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:12 smithi169 systemd[1]: Stopping Ceph prometheus.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:37:13.183 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:12 smithi169 bash[41525]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-prometheus.a 2023-12-16T07:37:13.183 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[40088]: ts=2023-12-16T07:37:13.016Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2023-12-16T07:37:13.183 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[40088]: ts=2023-12-16T07:37:13.016Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2023-12-16T07:37:13.184 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[40088]: ts=2023-12-16T07:37:13.016Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2023-12-16T07:37:13.184 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[40088]: ts=2023-12-16T07:37:13.016Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2023-12-16T07:37:13.184 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[40088]: ts=2023-12-16T07:37:13.016Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2023-12-16T07:37:13.184 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[40088]: ts=2023-12-16T07:37:13.016Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2023-12-16T07:37:13.184 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[40088]: ts=2023-12-16T07:37:13.017Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2023-12-16T07:37:13.184 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[40088]: ts=2023-12-16T07:37:13.017Z caller=main.go:828 level=info msg="Scrape manager stopped" 2023-12-16T07:37:13.184 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[40088]: ts=2023-12-16T07:37:13.017Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2023-12-16T07:37:13.184 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[40088]: ts=2023-12-16T07:37:13.018Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2023-12-16T07:37:13.184 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[40088]: ts=2023-12-16T07:37:13.018Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2023-12-16T07:37:13.184 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[40088]: ts=2023-12-16T07:37:13.018Z caller=main.go:1066 level=info msg="See you next time!" 2023-12-16T07:37:13.513 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[25083]: cluster 2023-12-16T07:37:12.207857+0000 mgr.y (mgr.24367) 65 : cluster [DBG] pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:13.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:13 smithi007 bash[25009]: cluster 2023-12-16T07:37:12.207857+0000 mgr.y (mgr.24367) 65 : cluster [DBG] pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:13.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:13 smithi007 bash[30436]: cluster 2023-12-16T07:37:12.207857+0000 mgr.y (mgr.24367) 65 : cluster [DBG] pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:13.778 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[41535]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-prometheus-a 2023-12-16T07:37:13.778 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 bash[41571]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-prometheus.a 2023-12-16T07:37:13.778 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@prometheus.a.service: Succeeded. 2023-12-16T07:37:13.778 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 systemd[1]: Stopped Ceph prometheus.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:37:13.778 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:13 smithi169 systemd[1]: Started Ceph prometheus.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:37:14.439 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574910 got 55834574914 for osd.1 2023-12-16T07:37:14.439 DEBUG:teuthology.parallel:result is None 2023-12-16T07:37:15.006 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[41600]: ts=2023-12-16T07:37:14.844Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2023-12-16T07:37:15.007 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[41600]: ts=2023-12-16T07:37:14.844Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2023-12-16T07:37:15.007 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[41600]: ts=2023-12-16T07:37:14.844Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2023-12-16T07:37:15.007 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[41600]: ts=2023-12-16T07:37:14.844Z caller=main.go:518 level=info host_details="(Linux 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 smithi169 (none))" 2023-12-16T07:37:15.007 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[41600]: ts=2023-12-16T07:37:14.844Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2023-12-16T07:37:15.007 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[41600]: ts=2023-12-16T07:37:14.844Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2023-12-16T07:37:15.007 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[41600]: ts=2023-12-16T07:37:14.847Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2023-12-16T07:37:15.007 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[41600]: ts=2023-12-16T07:37:14.847Z caller=main.go:923 level=info msg="Starting TSDB ..." 2023-12-16T07:37:15.008 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[41600]: ts=2023-12-16T07:37:14.851Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2023-12-16T07:37:15.008 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[41600]: ts=2023-12-16T07:37:14.856Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2023-12-16T07:37:15.008 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[41600]: ts=2023-12-16T07:37:14.856Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.719µs 2023-12-16T07:37:15.008 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[41600]: ts=2023-12-16T07:37:14.856Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2023-12-16T07:37:15.008 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:15 smithi169 bash[41600]: ts=2023-12-16T07:37:15.004Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2023-12-16T07:37:15.008 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:15 smithi169 bash[41600]: ts=2023-12-16T07:37:15.005Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2023-12-16T07:37:15.009 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.764721+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:15.009 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.768989+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T07:37:15.009 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.769512+0000 mgr.y (mgr.24367) 66 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T07:37:15.009 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.771005+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2023-12-16T07:37:15.009 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.771407+0000 mgr.y (mgr.24367) 67 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2023-12-16T07:37:15.010 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.778182+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:15.010 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.785112+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:37:15.010 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.785730+0000 mgr.y (mgr.24367) 68 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:37:15.010 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.798791+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.169:9095"}]: dispatch 2023-12-16T07:37:15.010 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.799521+0000 mgr.y (mgr.24367) 69 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.169:9095"}]: dispatch 2023-12-16T07:37:15.010 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.805276+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:15.011 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.812990+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T07:37:15.011 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.813758+0000 mgr.y (mgr.24367) 70 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T07:37:15.011 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.823082+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.169:3000"}]: dispatch 2023-12-16T07:37:15.011 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.823576+0000 mgr.y (mgr.24367) 71 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.169:3000"}]: dispatch 2023-12-16T07:37:15.011 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.829867+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:15.011 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.833209+0000 mon.a (mon.0) 676 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:37:15.012 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.835824+0000 mon.a (mon.0) 677 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:37:15.012 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:14 smithi169 bash[25083]: audit 2023-12-16T07:37:13.837476+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:37:15.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.764721+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:15.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.768989+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T07:37:15.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.769512+0000 mgr.y (mgr.24367) 66 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T07:37:15.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.771005+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2023-12-16T07:37:15.124 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.771407+0000 mgr.y (mgr.24367) 67 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2023-12-16T07:37:15.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.778182+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:15.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.785112+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:37:15.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.785730+0000 mgr.y (mgr.24367) 68 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:37:15.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.798791+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.169:9095"}]: dispatch 2023-12-16T07:37:15.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.799521+0000 mgr.y (mgr.24367) 69 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.169:9095"}]: dispatch 2023-12-16T07:37:15.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.805276+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:15.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.812990+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T07:37:15.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.813758+0000 mgr.y (mgr.24367) 70 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T07:37:15.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.823082+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.169:3000"}]: dispatch 2023-12-16T07:37:15.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.823576+0000 mgr.y (mgr.24367) 71 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.169:3000"}]: dispatch 2023-12-16T07:37:15.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.829867+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:15.125 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.833209+0000 mon.a (mon.0) 676 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:37:15.126 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.835824+0000 mon.a (mon.0) 677 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:37:15.126 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[30436]: audit 2023-12-16T07:37:13.837476+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:37:15.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.764721+0000 mon.a (mon.0) 666 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:15.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.768989+0000 mon.a (mon.0) 667 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T07:37:15.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.769512+0000 mgr.y (mgr.24367) 66 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T07:37:15.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.771005+0000 mon.a (mon.0) 668 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2023-12-16T07:37:15.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.771407+0000 mgr.y (mgr.24367) 67 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2023-12-16T07:37:15.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.778182+0000 mon.a (mon.0) 669 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:15.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.785112+0000 mon.a (mon.0) 670 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:37:15.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.785730+0000 mgr.y (mgr.24367) 68 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:37:15.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.798791+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.169:9095"}]: dispatch 2023-12-16T07:37:15.126 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.799521+0000 mgr.y (mgr.24367) 69 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.169:9095"}]: dispatch 2023-12-16T07:37:15.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.805276+0000 mon.a (mon.0) 672 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:15.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.812990+0000 mon.a (mon.0) 673 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T07:37:15.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.813758+0000 mgr.y (mgr.24367) 70 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T07:37:15.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.823082+0000 mon.a (mon.0) 674 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.169:3000"}]: dispatch 2023-12-16T07:37:15.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.823576+0000 mgr.y (mgr.24367) 71 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.169:3000"}]: dispatch 2023-12-16T07:37:15.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.829867+0000 mon.a (mon.0) 675 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:15.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.833209+0000 mon.a (mon.0) 676 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:37:15.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.835824+0000 mon.a (mon.0) 677 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:37:15.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:14 smithi007 bash[25009]: audit 2023-12-16T07:37:13.837476+0000 mon.a (mon.0) 678 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:37:15.271 INFO:teuthology.orchestra.run.smithi007.stdout:231928234007 2023-12-16T07:37:15.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:15 smithi169 bash[41600]: ts=2023-12-16T07:37:15.005Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=54.831µs wal_replay_duration=148.437244ms total_replay_duration=148.519126ms 2023-12-16T07:37:15.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:15 smithi169 bash[41600]: ts=2023-12-16T07:37:15.009Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2023-12-16T07:37:15.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:15 smithi169 bash[41600]: ts=2023-12-16T07:37:15.009Z caller=main.go:947 level=info msg="TSDB started" 2023-12-16T07:37:15.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:15 smithi169 bash[41600]: ts=2023-12-16T07:37:15.009Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-12-16T07:37:15.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:15 smithi169 bash[41600]: ts=2023-12-16T07:37:15.043Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=34.329285ms db_storage=1.356µs remote_storage=2.176µs web_handler=585ns query_engine=1.31µs scrape=559.153µs scrape_sd=56.169µs notify=60.539µs notify_sd=17.964µs rules=33.078702ms 2023-12-16T07:37:15.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:37:15 smithi169 bash[41600]: ts=2023-12-16T07:37:15.043Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2023-12-16T07:37:15.956 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495642 got 197568495646 for osd.6 2023-12-16T07:37:15.956 DEBUG:teuthology.parallel:result is None 2023-12-16T07:37:16.122 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:15 smithi007 bash[30436]: cluster 2023-12-16T07:37:14.208492+0000 mgr.y (mgr.24367) 72 : cluster [DBG] pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:16.122 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:15 smithi007 bash[30436]: audit 2023-12-16T07:37:15.269425+0000 mon.a (mon.0) 679 : audit [DBG] from='client.? 172.21.15.7:0/2156964368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-12-16T07:37:16.122 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:15 smithi007 bash[25009]: cluster 2023-12-16T07:37:14.208492+0000 mgr.y (mgr.24367) 72 : cluster [DBG] pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:16.122 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:15 smithi007 bash[25009]: audit 2023-12-16T07:37:15.269425+0000 mon.a (mon.0) 679 : audit [DBG] from='client.? 172.21.15.7:0/2156964368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-12-16T07:37:16.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:15 smithi169 bash[25083]: cluster 2023-12-16T07:37:14.208492+0000 mgr.y (mgr.24367) 72 : cluster [DBG] pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:16.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:15 smithi169 bash[25083]: audit 2023-12-16T07:37:15.269425+0000 mon.a (mon.0) 679 : audit [DBG] from='client.? 172.21.15.7:0/2156964368' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-12-16T07:37:16.476 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:16 smithi007 bash[49264]: level=info ts=2023-12-16T07:37:16.120Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2023-12-16T07:37:16.476 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:16 smithi007 bash[49264]: level=info ts=2023-12-16T07:37:16.120Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2023-12-16T07:37:16.476 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:16 smithi007 bash[49264]: level=info ts=2023-12-16T07:37:16.124Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.7 port=9094 2023-12-16T07:37:16.476 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:16 smithi007 bash[49264]: level=info ts=2023-12-16T07:37:16.125Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2023-12-16T07:37:16.476 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:16 smithi007 bash[49264]: level=info ts=2023-12-16T07:37:16.177Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2023-12-16T07:37:16.476 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:16 smithi007 bash[49264]: level=info ts=2023-12-16T07:37:16.177Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2023-12-16T07:37:16.476 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:16 smithi007 bash[49264]: level=info ts=2023-12-16T07:37:16.182Z caller=main.go:518 msg=Listening address=:9093 2023-12-16T07:37:16.477 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:16 smithi007 bash[49264]: level=info ts=2023-12-16T07:37:16.183Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2023-12-16T07:37:16.656 INFO:teuthology.orchestra.run.smithi007.stdout:34359738443 2023-12-16T07:37:16.863 INFO:teuthology.orchestra.run.smithi007.stdout:163208757287 2023-12-16T07:37:17.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:16 smithi169 bash[25083]: audit 2023-12-16T07:37:16.655030+0000 mon.a (mon.0) 680 : audit [DBG] from='client.? 172.21.15.7:0/26606707' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-12-16T07:37:17.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:16 smithi007 bash[25009]: audit 2023-12-16T07:37:16.655030+0000 mon.a (mon.0) 680 : audit [DBG] from='client.? 172.21.15.7:0/26606707' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-12-16T07:37:17.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:16 smithi007 bash[30436]: audit 2023-12-16T07:37:16.655030+0000 mon.a (mon.0) 680 : audit [DBG] from='client.? 172.21.15.7:0/26606707' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-12-16T07:37:17.921 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234003 got 231928234007 for osd.7 2023-12-16T07:37:17.921 DEBUG:teuthology.parallel:result is None 2023-12-16T07:37:17.932 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:17 smithi169 bash[25083]: cluster 2023-12-16T07:37:16.209057+0000 mgr.y (mgr.24367) 73 : cluster [DBG] pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:17.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:17 smithi169 bash[25083]: audit 2023-12-16T07:37:16.861858+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.7:0/1533765176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-12-16T07:37:17.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:17 smithi169 bash[25083]: audit 2023-12-16T07:37:17.679982+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:18.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:17 smithi007 bash[25009]: cluster 2023-12-16T07:37:16.209057+0000 mgr.y (mgr.24367) 73 : cluster [DBG] pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:18.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:17 smithi007 bash[25009]: audit 2023-12-16T07:37:16.861858+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.7:0/1533765176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-12-16T07:37:18.127 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:17 smithi007 bash[25009]: audit 2023-12-16T07:37:17.679982+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:18.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:17 smithi007 bash[30436]: cluster 2023-12-16T07:37:16.209057+0000 mgr.y (mgr.24367) 73 : cluster [DBG] pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:18.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:17 smithi007 bash[30436]: audit 2023-12-16T07:37:16.861858+0000 mon.c (mon.1) 14 : audit [DBG] from='client.? 172.21.15.7:0/1533765176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-12-16T07:37:18.128 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:17 smithi007 bash[30436]: audit 2023-12-16T07:37:17.679982+0000 mon.a (mon.0) 681 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:18.162 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738439 got 34359738443 for osd.0 2023-12-16T07:37:18.163 DEBUG:teuthology.parallel:result is None 2023-12-16T07:37:18.566 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:18 smithi007 bash[49264]: level=info ts=2023-12-16T07:37:18.125Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000286096s 2023-12-16T07:37:18.981 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757282 got 163208757287 for osd.5 2023-12-16T07:37:18.981 DEBUG:teuthology.parallel:result is None 2023-12-16T07:37:19.126 INFO:teuthology.orchestra.run.smithi007.stdout:77309411388 2023-12-16T07:37:19.160 INFO:teuthology.orchestra.run.smithi007.stdout:128849018926 2023-12-16T07:37:19.200 INFO:teuthology.orchestra.run.smithi007.stdout:107374182452 2023-12-16T07:37:20.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:19 smithi169 bash[25083]: cluster 2023-12-16T07:37:18.209760+0000 mgr.y (mgr.24367) 74 : cluster [DBG] pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:20.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:19 smithi169 bash[25083]: audit 2023-12-16T07:37:19.124708+0000 mon.a (mon.0) 682 : audit [DBG] from='client.? 172.21.15.7:0/1124218198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-12-16T07:37:20.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:19 smithi169 bash[25083]: audit 2023-12-16T07:37:19.159348+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.7:0/1445752044' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-12-16T07:37:20.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:19 smithi169 bash[25083]: audit 2023-12-16T07:37:19.199000+0000 mon.a (mon.0) 683 : audit [DBG] from='client.? 172.21.15.7:0/2859908859' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-12-16T07:37:20.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:19 smithi007 bash[25009]: cluster 2023-12-16T07:37:18.209760+0000 mgr.y (mgr.24367) 74 : cluster [DBG] pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:20.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:19 smithi007 bash[25009]: audit 2023-12-16T07:37:19.124708+0000 mon.a (mon.0) 682 : audit [DBG] from='client.? 172.21.15.7:0/1124218198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-12-16T07:37:20.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:19 smithi007 bash[25009]: audit 2023-12-16T07:37:19.159348+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.7:0/1445752044' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-12-16T07:37:20.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:19 smithi007 bash[25009]: audit 2023-12-16T07:37:19.199000+0000 mon.a (mon.0) 683 : audit [DBG] from='client.? 172.21.15.7:0/2859908859' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-12-16T07:37:20.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:19 smithi007 bash[30436]: cluster 2023-12-16T07:37:18.209760+0000 mgr.y (mgr.24367) 74 : cluster [DBG] pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:20.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:19 smithi007 bash[30436]: audit 2023-12-16T07:37:19.124708+0000 mon.a (mon.0) 682 : audit [DBG] from='client.? 172.21.15.7:0/1124218198' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-12-16T07:37:20.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:19 smithi007 bash[30436]: audit 2023-12-16T07:37:19.159348+0000 mon.c (mon.1) 15 : audit [DBG] from='client.? 172.21.15.7:0/1445752044' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-12-16T07:37:20.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:19 smithi007 bash[30436]: audit 2023-12-16T07:37:19.199000+0000 mon.a (mon.0) 683 : audit [DBG] from='client.? 172.21.15.7:0/2859908859' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-12-16T07:37:20.544 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018921 got 128849018926 for osd.4 2023-12-16T07:37:20.544 DEBUG:teuthology.parallel:result is None 2023-12-16T07:37:20.545 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411384 got 77309411388 for osd.2 2023-12-16T07:37:20.545 DEBUG:teuthology.parallel:result is None 2023-12-16T07:37:20.639 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182447 got 107374182452 for osd.3 2023-12-16T07:37:20.639 DEBUG:teuthology.parallel:result is None 2023-12-16T07:37:20.640 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-12-16T07:37:20.640 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph pg dump --format=json 2023-12-16T07:37:22.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:21 smithi169 bash[25083]: cluster 2023-12-16T07:37:20.210437+0000 mgr.y (mgr.24367) 75 : cluster [DBG] pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:22.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:21 smithi007 bash[25009]: cluster 2023-12-16T07:37:20.210437+0000 mgr.y (mgr.24367) 75 : cluster [DBG] pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:22.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:21 smithi007 bash[30436]: cluster 2023-12-16T07:37:20.210437+0000 mgr.y (mgr.24367) 75 : cluster [DBG] pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:22.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:37:22 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:37:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:37:23.138 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:37:23.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:22 smithi007 bash[30436]: audit 2023-12-16T07:37:21.964744+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:23.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:22 smithi007 bash[30436]: audit 2023-12-16T07:37:21.973607+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:23.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:22 smithi007 bash[30436]: audit 2023-12-16T07:37:21.980658+0000 mon.a (mon.0) 686 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:37:23.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:22 smithi007 bash[30436]: audit 2023-12-16T07:37:21.982654+0000 mon.a (mon.0) 687 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:37:23.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:22 smithi007 bash[30436]: audit 2023-12-16T07:37:21.983948+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:37:23.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:22 smithi007 bash[25009]: audit 2023-12-16T07:37:21.964744+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:23.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:22 smithi007 bash[25009]: audit 2023-12-16T07:37:21.973607+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:23.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:22 smithi007 bash[25009]: audit 2023-12-16T07:37:21.980658+0000 mon.a (mon.0) 686 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:37:23.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:22 smithi007 bash[25009]: audit 2023-12-16T07:37:21.982654+0000 mon.a (mon.0) 687 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:37:23.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:22 smithi007 bash[25009]: audit 2023-12-16T07:37:21.983948+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:37:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:22 smithi169 bash[25083]: audit 2023-12-16T07:37:21.964744+0000 mon.a (mon.0) 684 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:22 smithi169 bash[25083]: audit 2023-12-16T07:37:21.973607+0000 mon.a (mon.0) 685 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:22 smithi169 bash[25083]: audit 2023-12-16T07:37:21.980658+0000 mon.a (mon.0) 686 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:37:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:22 smithi169 bash[25083]: audit 2023-12-16T07:37:21.982654+0000 mon.a (mon.0) 687 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:37:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:22 smithi169 bash[25083]: audit 2023-12-16T07:37:21.983948+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:37:23.967 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:37:23 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:37:23] "GET /metrics HTTP/1.1" 200 191503 "" "Prometheus/2.33.4" 2023-12-16T07:37:23.968 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:23 smithi007 bash[25009]: cluster 2023-12-16T07:37:22.211107+0000 mgr.y (mgr.24367) 76 : cluster [DBG] pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:24.229 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:23 smithi007 bash[30436]: cluster 2023-12-16T07:37:22.211107+0000 mgr.y (mgr.24367) 76 : cluster [DBG] pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:23 smithi169 bash[25083]: cluster 2023-12-16T07:37:22.211107+0000 mgr.y (mgr.24367) 76 : cluster [DBG] pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:26.131 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:25 smithi007 bash[30436]: cluster 2023-12-16T07:37:24.212553+0000 mgr.y (mgr.24367) 77 : cluster [DBG] pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:26.131 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:25 smithi007 bash[30436]: audit 2023-12-16T07:37:25.777511+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:26.131 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:25 smithi007 bash[30436]: audit 2023-12-16T07:37:25.785892+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:26.131 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:25 smithi007 bash[25009]: cluster 2023-12-16T07:37:24.212553+0000 mgr.y (mgr.24367) 77 : cluster [DBG] pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:26.131 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:25 smithi007 bash[25009]: audit 2023-12-16T07:37:25.777511+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:26.131 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:25 smithi007 bash[25009]: audit 2023-12-16T07:37:25.785892+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:26.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:25 smithi169 bash[25083]: cluster 2023-12-16T07:37:24.212553+0000 mgr.y (mgr.24367) 77 : cluster [DBG] pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:26.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:25 smithi169 bash[25083]: audit 2023-12-16T07:37:25.777511+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:26.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:25 smithi169 bash[25083]: audit 2023-12-16T07:37:25.785892+0000 mon.a (mon.0) 690 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:26.247 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-16T07:37:26.250 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-12-16T07:37:26.425 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:37:26 smithi007 bash[49264]: level=info ts=2023-12-16T07:37:26.129Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003892314s 2023-12-16T07:37:27.006 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2023-12-16T07:37:26.212855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48900,"kb_used_data":6084,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912316,"statfs":{"total":767960285184,"available":767910211584,"internally_reserved":0,"allocated":6230016,"data_stored":3488071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004646"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-12-16T07:35:50.569579+0000","last_change":"2023-12-16T07:35:26.120325+0000","last_active":"2023-12-16T07:35:50.569579+0000","last_peered":"2023-12-16T07:35:50.569579+0000","last_clean":"2023-12-16T07:35:50.569579+0000","last_became_active":"2023-12-16T07:35:26.119387+0000","last_became_peered":"2023-12-16T07:35:26.119387+0000","last_unstale":"2023-12-16T07:35:50.569579+0000","last_undegraded":"2023-12-16T07:35:50.569579+0000","last_fullsized":"2023-12-16T07:35:50.569579+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T07:32:20.088930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T07:32:20.088930+0000","last_clean_scrub_stamp":"2023-12-16T07:32:20.088930+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T07:58:48.206730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234010,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6084,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739068,"statfs":{"total":95995035648,"available":95988805632,"internally_reserved":0,"allocated":1110016,"data_stored":766446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 07:37:24 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91300000000000003,"15min":0.91900000000000004},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2110000000000001,"5min":1.339,"15min":1.339},"last":0.747},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.91500000000000004,"15min":0.91700000000000004},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.258,"5min":1.393,"15min":1.393},"last":1.1020000000000001}]},{"osd":1,"last update":"Sat Dec 16 07:37:24 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.93400000000000005,"15min":0.94199999999999995},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":1.3480000000000001},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.95899999999999996,"15min":0.96999999999999997},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.79700000000000004}]},{"osd":2,"last update":"Sat Dec 16 07:37:24 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.94799999999999995,"15min":0.94399999999999995},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":1.075},{"interface":"front","average":{"1min":1.006,"5min":0.98999999999999999,"15min":0.98799999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":1.256}]},{"osd":3,"last update":"Sat Dec 16 07:37:24 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.075,"15min":1.089},"min":{"1min":0.75800000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.3120000000000001,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.0580000000000001,"15min":1.0740000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.98899999999999999}]},{"osd":4,"last update":"Sat Dec 16 07:37:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.0549999999999999,"15min":1.0700000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.1879999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1579999999999999,"15min":1.163},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":1.1970000000000001}]},{"osd":5,"last update":"Sat Dec 16 07:37:24 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.119,"15min":1.1259999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":1.169},{"interface":"front","average":{"1min":1.147,"5min":1.2150000000000001,"15min":1.2270000000000001},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.4990000000000001,"5min":1.516,"15min":1.516},"last":1.2350000000000001}]},{"osd":6,"last update":"Sat Dec 16 07:37:24 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.23,"15min":1.238},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.429,"5min":1.804,"15min":1.804},"last":1.304},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.2769999999999999,"15min":1.2849999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.323}]}]},{"osd":6,"up_from":46,"seq":197568495650,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.90800000000000003,"15min":0.90500000000000003},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.67200000000000004,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.2490000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.76300000000000001}]},{"osd":1,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.89700000000000002,"15min":0.89300000000000002},"min":{"1min":0.72599999999999998,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.282,"5min":1.484,"15min":1.484},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.95699999999999996,"15min":0.95299999999999996},"min":{"1min":0.71099999999999997,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":1.038}]},{"osd":2,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.96599999999999997,"15min":0.95999999999999996},"min":{"1min":0.63300000000000001,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.998,"15min":1.0029999999999999},"min":{"1min":0.64700000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.302,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":0.94499999999999995}]},{"osd":3,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.022,"15min":1.036},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.165,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.117,"5min":1.0640000000000001,"15min":1.0549999999999999},"min":{"1min":0.749,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.488,"5min":1.546,"15min":1.546},"last":1.0940000000000001}]},{"osd":4,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.96899999999999997,"15min":0.96699999999999997},"min":{"1min":0.68200000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.4059999999999999,"5min":1.73,"15min":1.73},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.022,"5min":1.0820000000000001,"15min":1.0920000000000001},"min":{"1min":0.68999999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.2529999999999999,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":0.82299999999999995}]},{"osd":5,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0589999999999999,"15min":1.0580000000000001},"min":{"1min":0.72699999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3580000000000001,"5min":1.629,"15min":1.629},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.052,"5min":1.1439999999999999,"15min":1.1599999999999999},"min":{"1min":0.81000000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.2789999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":0.92000000000000004}]},{"osd":7,"last update":"Sat Dec 16 07:36:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.784,"5min":1.784,"15min":1.784},"last":1.0149999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6332,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 07:36:49 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.69399999999999995,"15min":0.60399999999999998},"min":{"1min":0.56200000000000006,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.80100000000000005,"15min":0.67900000000000005},"min":{"1min":0.61799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":0.878}]},{"osd":2,"last update":"Sat Dec 16 07:36:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.83299999999999996,"15min":0.79600000000000004},"min":{"1min":0.57999999999999996,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.1220000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.78300000000000003,"15min":0.74199999999999999},"min":{"1min":0.55400000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.75800000000000001}]},{"osd":3,"last update":"Sat Dec 16 07:37:08 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.86199999999999999,"15min":0.82399999999999995},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.04,"5min":0.95899999999999996,"15min":0.90100000000000002},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.3460000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":1.0820000000000001}]},{"osd":4,"last update":"Sat Dec 16 07:36:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.05,"15min":1.0209999999999999},"min":{"1min":0.80600000000000005,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":1.028},{"interface":"front","average":{"1min":1.238,"5min":1.1240000000000001,"15min":1.0700000000000001},"min":{"1min":0.94999999999999996,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.647,"5min":1.647,"15min":1.647},"last":1.3009999999999999}]},{"osd":5,"last update":"Sat Dec 16 07:37:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.143,"15min":1.127},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.2250000000000001,"15min":1.2230000000000001},"min":{"1min":0.91500000000000004,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":1.4990000000000001}]},{"osd":6,"last update":"Sat Dec 16 07:36:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3440000000000001,"15min":1.3480000000000001},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":1.7330000000000001,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.3149999999999999,"15min":1.3080000000000001},"min":{"1min":1.0649999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":3.722,"5min":3.722,"15min":3.722},"last":1.264}]},{"osd":7,"last update":"Sat Dec 16 07:37:23 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":1.673,"5min":1.754,"15min":1.754},"last":1.429},{"interface":"front","average":{"1min":1.375,"5min":1.395,"15min":1.399},"min":{"1min":1.1759999999999999,"5min":1.131,"15min":1.131},"max":{"1min":1.629,"5min":1.841,"15min":1.841},"last":1.212}]}]},{"osd":0,"up_from":8,"seq":34359738446,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 07:36:51 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.70299999999999996,"15min":0.628},"min":{"1min":0.47999999999999998,"5min":0.432,"15min":0.432},"max":{"1min":1.0429999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.78400000000000003,"15min":0.69199999999999995},"min":{"1min":0.50800000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.3240000000000001,"5min":1.417,"15min":1.417},"last":0.63900000000000001}]},{"osd":2,"last update":"Sat Dec 16 07:36:27 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.754},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.361,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.872,"15min":0.80600000000000005},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.3919999999999999,"5min":2.0310000000000001,"15min":2.0310000000000001},"last":0.81499999999999995}]},{"osd":3,"last update":"Sat Dec 16 07:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91900000000000004,"15min":0.88600000000000001},"min":{"1min":0.60899999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.282,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.89200000000000002,"15min":0.87},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":0.91200000000000003}]},{"osd":4,"last update":"Sat Dec 16 07:36:36 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.0229999999999999,"15min":1.002},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.335,"5min":2.2080000000000002,"15min":2.2080000000000002},"last":1.264},{"interface":"front","average":{"1min":1.091,"5min":1.1279999999999999,"15min":1.097},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.512,"5min":2.161,"15min":2.161},"last":1.4339999999999999}]},{"osd":5,"last update":"Sat Dec 16 07:37:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1930000000000001,"15min":1.2},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.492,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1859999999999999,"15min":1.1759999999999999},"min":{"1min":0.81299999999999994,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.556,"5min":2.2669999999999999,"15min":2.2669999999999999},"last":1.107}]},{"osd":6,"last update":"Sat Dec 16 07:36:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.3280000000000001,"15min":1.3440000000000001},"min":{"1min":0.79000000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.6759999999999999,"5min":1.8759999999999999,"15min":1.8759999999999999},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.284,"5min":1.2809999999999999,"15min":1.2809999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":1.1339999999999999}]},{"osd":7,"last update":"Sat Dec 16 07:36:27 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.3859999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6268,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738884,"statfs":{"total":95995035648,"available":95988617216,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 07:36:32 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.72199999999999998,"15min":0.67900000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.77900000000000003,"15min":0.73699999999999999},"min":{"1min":0.502,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.141,"5min":1.321,"15min":1.321},"last":0.79400000000000004}]},{"osd":1,"last update":"Sat Dec 16 07:36:32 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.79300000000000004,"15min":0.73899999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.84299999999999997,"15min":0.77900000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.23,"5min":1.403,"15min":1.403},"last":0.88700000000000001}]},{"osd":3,"last update":"Sat Dec 16 07:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.90300000000000002,"15min":0.85199999999999998},"min":{"1min":0.55400000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":1.0840000000000001},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92600000000000005,"15min":0.91200000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.2270000000000001,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":1.113}]},{"osd":4,"last update":"Sat Dec 16 07:36:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.97499999999999998,"15min":0.95199999999999996},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.5190000000000001,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1100000000000001,"15min":1.091},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.429,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":1.2130000000000001}]},{"osd":5,"last update":"Sat Dec 16 07:37:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0469999999999999,"15min":1.0449999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3560000000000001,"5min":2.5939999999999999,"15min":2.5939999999999999},"last":1.024},{"interface":"front","average":{"1min":1.242,"5min":1.2010000000000001,"15min":1.175},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.6919999999999999,"5min":1.839,"15min":1.839},"last":1.4970000000000001}]},{"osd":6,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.3340000000000001,"15min":1.3500000000000001},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.641,"5min":1.7649999999999999,"15min":1.7649999999999999},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.278,"15min":1.282},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.534,"5min":1.78,"15min":1.78},"last":1.538}]},{"osd":7,"last update":"Sat Dec 16 07:36:25 2023","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":1.8080000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.367},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":2.7709999999999999,"5min":2.7709999999999999,"15min":2.7709999999999999},"last":1.4590000000000001}]}]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 07:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.71899999999999997,"15min":0.70899999999999996},"min":{"1min":0.53900000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0760000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.82099999999999995,"15min":0.78100000000000003},"min":{"1min":0.57099999999999995,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4039999999999999,"5min":1.696,"15min":1.696},"last":0.70099999999999996}]},{"osd":1,"last update":"Sat Dec 16 07:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.83599999999999997,"15min":0.81100000000000005},"min":{"1min":0.60199999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.196},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.86099999999999999,"15min":0.83499999999999996},"min":{"1min":0.66200000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.2949999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.79400000000000004}]},{"osd":2,"last update":"Sat Dec 16 07:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.91300000000000003,"15min":0.89900000000000002},"min":{"1min":0.68000000000000005,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.258,"5min":1.282,"15min":1.282},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.91300000000000003,"15min":0.89100000000000001},"min":{"1min":0.66400000000000003,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.3069999999999999,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.86899999999999999}]},{"osd":4,"last update":"Sat Dec 16 07:36:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.083,"15min":1.085},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.403,"5min":2.073,"15min":2.073},"last":1.05},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.042},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":1.417,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":1.081}]},{"osd":5,"last update":"Sat Dec 16 07:37:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.1950000000000001,"15min":1.1779999999999999},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.133},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.206,"15min":1.1859999999999999},"min":{"1min":0.91200000000000003,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.012}]},{"osd":6,"last update":"Sat Dec 16 07:36:54 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.218,"15min":1.22},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.97699999999999998,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.665,"5min":1.734,"15min":1.734},"last":1.228}]},{"osd":7,"last update":"Sat Dec 16 07:37:22 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.2749999999999999,"15min":1.2549999999999999},"min":{"1min":1.069,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.954,"5min":1.954,"15min":1.954},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.365,"5min":1.2969999999999999,"15min":1.2849999999999999},"min":{"1min":1.198,"5min":0.92700000000000005,"15min":0.92700000000000005},"max":{"1min":1.9039999999999999,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":1.2529999999999999}]}]},{"osd":4,"up_from":30,"seq":128849018928,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5756,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739396,"statfs":{"total":95995035648,"available":95989141504,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 07:36:36 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.86099999999999999,"15min":0.84299999999999997},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.2849999999999999,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.93799999999999994,"15min":0.94999999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2569999999999999,"5min":1.29,"15min":1.29},"last":0.76200000000000001}]},{"osd":1,"last update":"Sat Dec 16 07:36:36 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.875,"15min":0.875},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.224,"5min":1.266,"15min":1.266},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.93700000000000006,"15min":0.92900000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":0.877}]},{"osd":2,"last update":"Sat Dec 16 07:36:36 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.97599999999999998,"15min":0.97999999999999998},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.96899999999999997,"15min":0.96699999999999997},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.70699999999999996}]},{"osd":3,"last update":"Sat Dec 16 07:36:36 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.93899999999999995,"15min":0.93000000000000005},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.0089999999999999,"15min":1.008},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.228,"5min":1.452,"15min":1.452},"last":0.98599999999999999}]},{"osd":5,"last update":"Sat Dec 16 07:37:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.89700000000000002,"15min":0.88700000000000001},"min":{"1min":0.60499999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.206,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.0580000000000001,"15min":1.042},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.377,"5min":1.667,"15min":1.667},"last":1.0329999999999999}]},{"osd":6,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":1.0109999999999999,"15min":1.0269999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2869999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0489999999999999,"15min":1.0600000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":1.107}]},{"osd":7,"last update":"Sat Dec 16 07:36:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":1.1850000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757289,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 07:37:14 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.91100000000000003,"15min":0.88700000000000001},"min":{"1min":0.60799999999999998,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":1.012},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.91900000000000004,"15min":0.875},"min":{"1min":0.751,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.63400000000000001}]},{"osd":1,"last update":"Sat Dec 16 07:37:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":0.88500000000000001,"15min":0.82499999999999996},"min":{"1min":0.58199999999999996,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.4119999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.046,"5min":0.93400000000000005,"15min":0.91000000000000003},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":0.94699999999999995}]},{"osd":2,"last update":"Sat Dec 16 07:37:14 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.90900000000000003,"15min":0.89400000000000002},"min":{"1min":0.56799999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3200000000000001,"5min":1.423,"15min":1.423},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.94399999999999995,"15min":0.91500000000000004},"min":{"1min":0.71799999999999997,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":1.228}]},{"osd":3,"last update":"Sat Dec 16 07:37:14 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":0.96299999999999997,"15min":0.93500000000000005},"min":{"1min":0.65300000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.048,"5min":0.97999999999999998,"15min":0.95599999999999996},"min":{"1min":0.81100000000000005,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.2669999999999999,"5min":1.325,"15min":1.325},"last":1.0980000000000001}]},{"osd":4,"last update":"Sat Dec 16 07:37:14 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.83799999999999997,"15min":0.79200000000000004},"min":{"1min":0.48999999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":1.026,"5min":0.90100000000000002,"15min":0.84699999999999998},"min":{"1min":0.74299999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.82599999999999996}]},{"osd":6,"last update":"Sat Dec 16 07:36:46 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.95899999999999996,"15min":0.95499999999999996},"min":{"1min":0.55800000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.03,"5min":0.99399999999999999,"15min":0.98799999999999999},"min":{"1min":0.64400000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.89700000000000002}]},{"osd":7,"last update":"Sat Dec 16 07:36:26 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.593,"5min":1.593,"15min":1.593},"last":1.1639999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T07:37:27.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph pg dump --format=json 2023-12-16T07:37:28.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:27 smithi007 bash[25009]: cluster 2023-12-16T07:37:26.213163+0000 mgr.y (mgr.24367) 78 : cluster [DBG] pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:28.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:27 smithi007 bash[25009]: audit 2023-12-16T07:37:26.238649+0000 mgr.y (mgr.24367) 79 : audit [DBG] from='client.14631 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T07:37:28.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:27 smithi007 bash[30436]: cluster 2023-12-16T07:37:26.213163+0000 mgr.y (mgr.24367) 78 : cluster [DBG] pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:28.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:27 smithi007 bash[30436]: audit 2023-12-16T07:37:26.238649+0000 mgr.y (mgr.24367) 79 : audit [DBG] from='client.14631 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T07:37:28.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:27 smithi169 bash[25083]: cluster 2023-12-16T07:37:26.213163+0000 mgr.y (mgr.24367) 78 : cluster [DBG] pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:28.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:27 smithi169 bash[25083]: audit 2023-12-16T07:37:26.238649+0000 mgr.y (mgr.24367) 79 : audit [DBG] from='client.14631 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T07:37:29.749 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:37:30.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:29 smithi007 bash[25009]: cluster 2023-12-16T07:37:28.213819+0000 mgr.y (mgr.24367) 80 : cluster [DBG] pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:30.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:29 smithi007 bash[30436]: cluster 2023-12-16T07:37:28.213819+0000 mgr.y (mgr.24367) 80 : cluster [DBG] pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:30.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:29 smithi169 bash[25083]: cluster 2023-12-16T07:37:28.213819+0000 mgr.y (mgr.24367) 80 : cluster [DBG] pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:32.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:31 smithi007 bash[25009]: cluster 2023-12-16T07:37:30.214399+0000 mgr.y (mgr.24367) 81 : cluster [DBG] pgmap v53: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:32.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:31 smithi007 bash[30436]: cluster 2023-12-16T07:37:30.214399+0000 mgr.y (mgr.24367) 81 : cluster [DBG] pgmap v53: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:32.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:31 smithi169 bash[25083]: cluster 2023-12-16T07:37:30.214399+0000 mgr.y (mgr.24367) 81 : cluster [DBG] pgmap v53: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:32.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:37:32 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:37:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:37:32.888 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-16T07:37:32.891 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-12-16T07:37:33.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:33 smithi007 bash[25009]: cluster 2023-12-16T07:37:32.215042+0000 mgr.y (mgr.24367) 82 : cluster [DBG] pgmap v54: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:33.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:33 smithi007 bash[25009]: audit 2023-12-16T07:37:32.882565+0000 mgr.y (mgr.24367) 83 : audit [DBG] from='client.24550 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T07:37:33.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:33 smithi007 bash[30436]: cluster 2023-12-16T07:37:32.215042+0000 mgr.y (mgr.24367) 82 : cluster [DBG] pgmap v54: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:33.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:33 smithi007 bash[30436]: audit 2023-12-16T07:37:32.882565+0000 mgr.y (mgr.24367) 83 : audit [DBG] from='client.24550 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T07:37:33.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:33 smithi169 bash[25083]: cluster 2023-12-16T07:37:32.215042+0000 mgr.y (mgr.24367) 82 : cluster [DBG] pgmap v54: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:33.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:33 smithi169 bash[25083]: audit 2023-12-16T07:37:32.882565+0000 mgr.y (mgr.24367) 83 : audit [DBG] from='client.24550 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-16T07:37:33.709 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2023-12-16T07:37:32.214716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48900,"kb_used_data":6084,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912316,"statfs":{"total":767960285184,"available":767910211584,"internally_reserved":0,"allocated":6230016,"data_stored":3488071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004564"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-12-16T07:35:50.569579+0000","last_change":"2023-12-16T07:35:26.120325+0000","last_active":"2023-12-16T07:35:50.569579+0000","last_peered":"2023-12-16T07:35:50.569579+0000","last_clean":"2023-12-16T07:35:50.569579+0000","last_became_active":"2023-12-16T07:35:26.119387+0000","last_became_peered":"2023-12-16T07:35:26.119387+0000","last_unstale":"2023-12-16T07:35:50.569579+0000","last_undegraded":"2023-12-16T07:35:50.569579+0000","last_fullsized":"2023-12-16T07:35:50.569579+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-16T07:32:20.088930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-16T07:32:20.088930+0000","last_clean_scrub_stamp":"2023-12-16T07:32:20.088930+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-17T07:58:48.206730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234011,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6084,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739068,"statfs":{"total":95995035648,"available":95988805632,"internally_reserved":0,"allocated":1110016,"data_stored":766446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 07:37:24 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.91300000000000003,"15min":0.91900000000000004},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2110000000000001,"5min":1.339,"15min":1.339},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.91500000000000004,"15min":0.91700000000000004},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.258,"5min":1.393,"15min":1.393},"last":1.0169999999999999}]},{"osd":1,"last update":"Sat Dec 16 07:37:24 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.93400000000000005,"15min":0.94199999999999995},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.3480000000000001},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.95899999999999996,"15min":0.96999999999999997},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":1.365,"5min":1.365,"15min":1.365},"last":0.80900000000000005}]},{"osd":2,"last update":"Sat Dec 16 07:37:24 2023","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":0.94799999999999995,"15min":0.94399999999999995},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.006,"5min":0.98999999999999999,"15min":0.98799999999999999},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":1.3009999999999999}]},{"osd":3,"last update":"Sat Dec 16 07:37:24 2023","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.075,"15min":1.089},"min":{"1min":0.75800000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.3120000000000001,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.96199999999999997,"5min":1.0580000000000001,"15min":1.0740000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.93300000000000005}]},{"osd":4,"last update":"Sat Dec 16 07:37:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.0549999999999999,"15min":1.0700000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.1879999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.1579999999999999,"15min":1.163},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.98099999999999998}]},{"osd":5,"last update":"Sat Dec 16 07:37:24 2023","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.119,"15min":1.1259999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":1.147},{"interface":"front","average":{"1min":1.147,"5min":1.2150000000000001,"15min":1.2270000000000001},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.4990000000000001,"5min":1.516,"15min":1.516},"last":1.179}]},{"osd":6,"last update":"Sat Dec 16 07:37:24 2023","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.23,"15min":1.238},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.429,"5min":1.804,"15min":1.804},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.2769999999999999,"15min":1.2849999999999999},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":1.6890000000000001},"last":1.2669999999999999}]}]},{"osd":6,"up_from":46,"seq":197568495651,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.90800000000000003,"15min":0.90500000000000003},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.3280000000000001,"5min":1.3280000000000001,"15min":1.3280000000000001},"last":1},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"min":{"1min":0.67200000000000004,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.2490000000000001,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":1.3009999999999999}]},{"osd":1,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.89700000000000002,"15min":0.89300000000000002},"min":{"1min":0.72599999999999998,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.282,"5min":1.484,"15min":1.484},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.95699999999999996,"15min":0.95299999999999996},"min":{"1min":0.71099999999999997,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.3620000000000001,"5min":1.3620000000000001,"15min":1.3620000000000001},"last":1.22}]},{"osd":2,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":0.96599999999999997,"15min":0.95999999999999996},"min":{"1min":0.63300000000000001,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.998,"15min":1.0029999999999999},"min":{"1min":0.64700000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.302,"5min":1.4870000000000001,"15min":1.4870000000000001},"last":1.085}]},{"osd":3,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":1.022,"15min":1.036},"min":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.165,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.117,"5min":1.0640000000000001,"15min":1.0549999999999999},"min":{"1min":0.749,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.488,"5min":1.546,"15min":1.546},"last":1.121}]},{"osd":4,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.96899999999999997,"15min":0.96699999999999997},"min":{"1min":0.68200000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.4059999999999999,"5min":1.73,"15min":1.73},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.022,"5min":1.0820000000000001,"15min":1.0920000000000001},"min":{"1min":0.68999999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.2529999999999999,"5min":2.4620000000000002,"15min":2.4620000000000002},"last":1.0649999999999999}]},{"osd":5,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0589999999999999,"15min":1.0580000000000001},"min":{"1min":0.72699999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.3580000000000001,"5min":1.629,"15min":1.629},"last":1.244},{"interface":"front","average":{"1min":1.052,"5min":1.1439999999999999,"15min":1.1599999999999999},"min":{"1min":0.81000000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.2789999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":1.1839999999999999}]},{"osd":7,"last update":"Sat Dec 16 07:37:28 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.1759999999999999,"15min":1.1819999999999999},"min":{"1min":0.78900000000000003,"5min":0.751,"15min":0.751},"max":{"1min":1.431,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.286},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.24,"15min":1.244},"min":{"1min":1.0149999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.504,"5min":1.784,"15min":1.784},"last":1.6579999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6332,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 07:36:49 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.69399999999999995,"15min":0.60399999999999998},"min":{"1min":0.56200000000000006,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":2.1190000000000002,"5min":2.1190000000000002,"15min":2.1190000000000002},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.80100000000000005,"15min":0.67900000000000005},"min":{"1min":0.61799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.657,"5min":1.657,"15min":1.657},"last":1.2}]},{"osd":2,"last update":"Sat Dec 16 07:37:28 2023","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.87,"15min":0.80800000000000005},"min":{"1min":0.65700000000000003,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":1.026},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79700000000000004,"15min":0.747},"min":{"1min":0.56999999999999995,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.3029999999999999,"5min":1.3029999999999999,"15min":1.3029999999999999},"last":0.73399999999999999}]},{"osd":3,"last update":"Sat Dec 16 07:37:08 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.86199999999999999,"15min":0.82399999999999995},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.8129999999999999,"5min":1.8129999999999999,"15min":1.8129999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.04,"5min":0.95899999999999996,"15min":0.90100000000000002},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.3460000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.83699999999999997}]},{"osd":4,"last update":"Sat Dec 16 07:36:33 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.05,"15min":1.0209999999999999},"min":{"1min":0.80600000000000005,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":1.149},{"interface":"front","average":{"1min":1.238,"5min":1.1240000000000001,"15min":1.0700000000000001},"min":{"1min":0.94999999999999996,"5min":0.94299999999999995,"15min":0.94299999999999995},"max":{"1min":1.647,"5min":1.647,"15min":1.647},"last":1.1120000000000001}]},{"osd":5,"last update":"Sat Dec 16 07:37:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.143,"15min":1.127},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"max":{"1min":3.7730000000000001,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.2250000000000001,"15min":1.2230000000000001},"min":{"1min":0.91500000000000004,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":1.7589999999999999},"last":1.0760000000000001}]},{"osd":6,"last update":"Sat Dec 16 07:36:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3440000000000001,"15min":1.3480000000000001},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"max":{"1min":1.7330000000000001,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":1.355},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.3149999999999999,"15min":1.3080000000000001},"min":{"1min":1.0649999999999999,"5min":0.96899999999999997,"15min":0.96899999999999997},"max":{"1min":3.722,"5min":3.722,"15min":3.722},"last":1.2669999999999999}]},{"osd":7,"last update":"Sat Dec 16 07:37:23 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":1.673,"5min":1.754,"15min":1.754},"last":1.228},{"interface":"front","average":{"1min":1.375,"5min":1.395,"15min":1.399},"min":{"1min":1.1759999999999999,"5min":1.131,"15min":1.131},"max":{"1min":1.629,"5min":1.841,"15min":1.841},"last":1.2869999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Dec 16 07:36:51 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.70299999999999996,"15min":0.628},"min":{"1min":0.47999999999999998,"5min":0.432,"15min":0.432},"max":{"1min":1.0429999999999999,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.624},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.78400000000000003,"15min":0.69199999999999995},"min":{"1min":0.50800000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.3240000000000001,"5min":1.417,"15min":1.417},"last":0.68300000000000005}]},{"osd":2,"last update":"Sat Dec 16 07:36:27 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.754},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.361,"5min":2.1150000000000002,"15min":2.1150000000000002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.872,"15min":0.80600000000000005},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.3919999999999999,"5min":2.0310000000000001,"15min":2.0310000000000001},"last":0.90900000000000003}]},{"osd":3,"last update":"Sat Dec 16 07:37:05 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.91900000000000004,"15min":0.88600000000000001},"min":{"1min":0.60899999999999999,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.282,"5min":1.5389999999999999,"15min":1.5389999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.89200000000000002,"15min":0.87},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.6899999999999999,"5min":1.6899999999999999,"15min":1.6899999999999999},"last":1.008}]},{"osd":4,"last update":"Sat Dec 16 07:36:36 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.0229999999999999,"15min":1.002},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":1.335,"5min":2.2080000000000002,"15min":2.2080000000000002},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.091,"5min":1.1279999999999999,"15min":1.097},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.512,"5min":2.161,"15min":2.161},"last":1.22}]},{"osd":5,"last update":"Sat Dec 16 07:37:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1930000000000001,"15min":1.2},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.492,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":1.25},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1859999999999999,"15min":1.1759999999999999},"min":{"1min":0.81299999999999994,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.556,"5min":2.2669999999999999,"15min":2.2669999999999999},"last":1.022}]},{"osd":6,"last update":"Sat Dec 16 07:36:51 2023","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.3280000000000001,"15min":1.3440000000000001},"min":{"1min":0.79000000000000004,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.6759999999999999,"5min":1.8759999999999999,"15min":1.8759999999999999},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.284,"5min":1.2809999999999999,"15min":1.2809999999999999},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":1.8540000000000001,"5min":1.8540000000000001,"15min":1.8540000000000001},"last":1.355}]},{"osd":7,"last update":"Sat Dec 16 07:36:27 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":1.427},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.123}]}]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6268,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738884,"statfs":{"total":95995035648,"available":95988617216,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 07:36:32 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.72199999999999998,"15min":0.67900000000000005},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.77900000000000003,"15min":0.73699999999999999},"min":{"1min":0.502,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.141,"5min":1.321,"15min":1.321},"last":0.86399999999999999}]},{"osd":1,"last update":"Sat Dec 16 07:36:32 2023","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.79300000000000004,"15min":0.73899999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.84299999999999997,"15min":0.77900000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.23,"5min":1.403,"15min":1.403},"last":0.94299999999999995}]},{"osd":3,"last update":"Sat Dec 16 07:37:07 2023","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.90300000000000002,"15min":0.85199999999999998},"min":{"1min":0.55400000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":1.3200000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92600000000000005,"15min":0.91200000000000003},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.2270000000000001,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.98499999999999999}]},{"osd":4,"last update":"Sat Dec 16 07:36:37 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.97499999999999998,"15min":0.95199999999999996},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.5190000000000001,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.1160000000000001,"5min":1.1100000000000001,"15min":1.091},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.429,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":1.278}]},{"osd":5,"last update":"Sat Dec 16 07:37:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0469999999999999,"15min":1.0449999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.3560000000000001,"5min":2.5939999999999999,"15min":2.5939999999999999},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.242,"5min":1.2010000000000001,"15min":1.175},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.6919999999999999,"5min":1.839,"15min":1.839},"last":1.4970000000000001}]},{"osd":6,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.3340000000000001,"15min":1.3500000000000001},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":1.641,"5min":1.7649999999999999,"15min":1.7649999999999999},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.278,"15min":1.282},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.534,"5min":1.78,"15min":1.78},"last":1.353}]},{"osd":7,"last update":"Sat Dec 16 07:37:26 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.397,"15min":1.4119999999999999},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.5589999999999999,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":1.425},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.397,"15min":1.4119999999999999},"min":{"1min":0.81799999999999995,"5min":0.745,"15min":0.745},"max":{"1min":1.6679999999999999,"5min":2.7709999999999999,"15min":2.7709999999999999},"last":1.4419999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 07:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.71899999999999997,"15min":0.70899999999999996},"min":{"1min":0.53900000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.0760000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.874},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.82099999999999995,"15min":0.78100000000000003},"min":{"1min":0.57099999999999995,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.4039999999999999,"5min":1.696,"15min":1.696},"last":0.80000000000000004}]},{"osd":1,"last update":"Sat Dec 16 07:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.83599999999999997,"15min":0.81100000000000005},"min":{"1min":0.60199999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"last":1.4610000000000001},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.86099999999999999,"15min":0.83499999999999996},"min":{"1min":0.66200000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.2949999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":1.593}]},{"osd":2,"last update":"Sat Dec 16 07:37:04 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.91300000000000003,"15min":0.89900000000000002},"min":{"1min":0.68000000000000005,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.258,"5min":1.282,"15min":1.282},"last":1.6699999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.91300000000000003,"15min":0.89100000000000001},"min":{"1min":0.66400000000000003,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.3069999999999999,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.95999999999999996}]},{"osd":4,"last update":"Sat Dec 16 07:36:35 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.083,"15min":1.085},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.403,"5min":2.073,"15min":2.073},"last":1.091},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.042},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":1.417,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":2.0270000000000001}]},{"osd":5,"last update":"Sat Dec 16 07:37:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.1950000000000001,"15min":1.1779999999999999},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.3079999999999998,"5min":2.3079999999999998,"15min":2.3079999999999998},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.206,"15min":1.1859999999999999},"min":{"1min":0.91200000000000003,"5min":0.90300000000000002,"15min":0.90300000000000002},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":0.93799999999999994}]},{"osd":6,"last update":"Sat Dec 16 07:36:54 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.218,"15min":1.22},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":2.1019999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.97699999999999998,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.665,"5min":1.734,"15min":1.734},"last":1.02}]},{"osd":7,"last update":"Sat Dec 16 07:37:22 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.2749999999999999,"15min":1.2549999999999999},"min":{"1min":1.069,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":1.954,"5min":1.954,"15min":1.954},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.2969999999999999,"15min":1.2849999999999999},"min":{"1min":1.198,"5min":0.92700000000000005,"15min":0.92700000000000005},"max":{"1min":1.9039999999999999,"5min":1.9039999999999999,"15min":1.9039999999999999},"last":2.1859999999999999}]}]},{"osd":4,"up_from":30,"seq":128849018929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5756,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739396,"statfs":{"total":95995035648,"available":95989141504,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 07:36:36 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.86099999999999999,"15min":0.84299999999999997},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.2849999999999999,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.93799999999999994,"15min":0.94999999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.2569999999999999,"5min":1.29,"15min":1.29},"last":0.66300000000000003}]},{"osd":1,"last update":"Sat Dec 16 07:36:36 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.875,"15min":0.875},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.224,"5min":1.266,"15min":1.266},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.93700000000000006,"15min":0.92900000000000005},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.444,"5min":1.444,"15min":1.444},"last":0.748}]},{"osd":2,"last update":"Sat Dec 16 07:36:36 2023","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":0.97599999999999998,"15min":0.97999999999999998},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.96899999999999997,"15min":0.96699999999999997},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.282,"5min":1.282,"15min":1.282},"last":0.77900000000000003}]},{"osd":3,"last update":"Sat Dec 16 07:36:36 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.93899999999999995,"15min":0.93000000000000005},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.0089999999999999,"15min":1.008},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.228,"5min":1.452,"15min":1.452},"last":0.72099999999999997}]},{"osd":5,"last update":"Sat Dec 16 07:37:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":0.89700000000000002,"15min":0.88700000000000001},"min":{"1min":0.60499999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.206,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.0580000000000001,"15min":1.042},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.377,"5min":1.667,"15min":1.667},"last":0.58699999999999997}]},{"osd":6,"last update":"Sat Dec 16 07:36:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":1.0109999999999999,"15min":1.0269999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2869999999999999,"5min":1.3400000000000001,"15min":1.3400000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.0489999999999999,"15min":1.0600000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.736,"5min":1.736,"15min":1.736},"last":0.70299999999999996}]},{"osd":7,"last update":"Sat Dec 16 07:37:28 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.077,"15min":1.085},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1359999999999999,"15min":1.1439999999999999},"min":{"1min":0.66100000000000003,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.4239999999999999,"5min":1.599,"15min":1.599},"last":0.76600000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757290,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Dec 16 07:37:14 2023","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.91100000000000003,"15min":0.88700000000000001},"min":{"1min":0.60799999999999998,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.91900000000000004,"15min":0.875},"min":{"1min":0.751,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.3340000000000001}]},{"osd":1,"last update":"Sat Dec 16 07:37:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":0.88500000000000001,"15min":0.82499999999999996},"min":{"1min":0.58199999999999996,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.4119999999999999,"5min":2.4119999999999999,"15min":2.4119999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":1.046,"5min":0.93400000000000005,"15min":0.91000000000000003},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":2.4260000000000002},"last":1.429}]},{"osd":2,"last update":"Sat Dec 16 07:37:14 2023","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.90900000000000003,"15min":0.89400000000000002},"min":{"1min":0.56799999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.3200000000000001,"5min":1.423,"15min":1.423},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.94399999999999995,"15min":0.91500000000000004},"min":{"1min":0.71799999999999997,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":1.492,"5min":1.492,"15min":1.492},"last":1.0680000000000001}]},{"osd":3,"last update":"Sat Dec 16 07:37:14 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":0.96299999999999997,"15min":0.93500000000000005},"min":{"1min":0.65300000000000002,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":1.052},{"interface":"front","average":{"1min":1.048,"5min":0.97999999999999998,"15min":0.95599999999999996},"min":{"1min":0.81100000000000005,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.2669999999999999,"5min":1.325,"15min":1.325},"last":1.1890000000000001}]},{"osd":4,"last update":"Sat Dec 16 07:37:14 2023","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":0.83799999999999997,"15min":0.79200000000000004},"min":{"1min":0.48999999999999999,"5min":0.437,"15min":0.437},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.026,"5min":0.90100000000000002,"15min":0.84699999999999998},"min":{"1min":0.74299999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.1299999999999999}]},{"osd":6,"last update":"Sat Dec 16 07:36:46 2023","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.95899999999999996,"15min":0.95499999999999996},"min":{"1min":0.55800000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.03,"5min":0.99399999999999999,"15min":0.98799999999999999},"min":{"1min":0.64400000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":1.246}]},{"osd":7,"last update":"Sat Dec 16 07:37:27 2023","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.123,"15min":1.123},"min":{"1min":0.80200000000000005,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.3999999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":1.373},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.1799999999999999,"15min":1.1759999999999999},"min":{"1min":0.67000000000000004,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":1.278}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-16T07:37:33.710 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-12-16T07:37:33.711 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-16T07:37:33.711 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-12-16T07:37:33.711 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- ceph health --format=json 2023-12-16T07:37:34.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:37:33 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:37:33] "GET /metrics HTTP/1.1" 200 191503 "" "Prometheus/2.33.4" 2023-12-16T07:37:35.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:35 smithi007 bash[25009]: cluster 2023-12-16T07:37:34.215659+0000 mgr.y (mgr.24367) 84 : cluster [DBG] pgmap v55: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:35.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:35 smithi007 bash[30436]: cluster 2023-12-16T07:37:34.215659+0000 mgr.y (mgr.24367) 84 : cluster [DBG] pgmap v55: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:35 smithi169 bash[25083]: cluster 2023-12-16T07:37:34.215659+0000 mgr.y (mgr.24367) 84 : cluster [DBG] pgmap v55: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:36.438 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/mon.a/config 2023-12-16T07:37:37.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:37 smithi007 bash[25009]: cluster 2023-12-16T07:37:36.216312+0000 mgr.y (mgr.24367) 85 : cluster [DBG] pgmap v56: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:37.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:37 smithi007 bash[30436]: cluster 2023-12-16T07:37:36.216312+0000 mgr.y (mgr.24367) 85 : cluster [DBG] pgmap v56: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:37.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:37 smithi169 bash[25083]: cluster 2023-12-16T07:37:36.216312+0000 mgr.y (mgr.24367) 85 : cluster [DBG] pgmap v56: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:39.483 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-12-16T07:37:39.483 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-16T07:37:39.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:39 smithi007 bash[25009]: cluster 2023-12-16T07:37:38.216913+0000 mgr.y (mgr.24367) 86 : cluster [DBG] pgmap v57: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:39.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:39 smithi007 bash[30436]: cluster 2023-12-16T07:37:38.216913+0000 mgr.y (mgr.24367) 86 : cluster [DBG] pgmap v57: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:39.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:39 smithi169 bash[25083]: cluster 2023-12-16T07:37:38.216913+0000 mgr.y (mgr.24367) 86 : cluster [DBG] pgmap v57: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:40.190 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-12-16T07:37:40.191 INFO:tasks.cephadm:Setup complete, yielding 2023-12-16T07:37:40.191 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T07:37:40.202 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2023-12-16T07:37:40.203 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2023-12-16T07:37:40.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:40 smithi007 bash[30436]: audit 2023-12-16T07:37:39.481533+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.7:0/1402887475' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-16T07:37:40.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:40 smithi007 bash[25009]: audit 2023-12-16T07:37:39.481533+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.7:0/1402887475' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-16T07:37:40.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:40 smithi169 bash[25083]: audit 2023-12-16T07:37:39.481533+0000 mon.c (mon.1) 16 : audit [DBG] from='client.? 172.21.15.7:0/1402887475' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-12-16T07:37:41.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:41 smithi007 bash[25009]: cluster 2023-12-16T07:37:40.217495+0000 mgr.y (mgr.24367) 87 : cluster [DBG] pgmap v58: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:41.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:41 smithi007 bash[30436]: cluster 2023-12-16T07:37:40.217495+0000 mgr.y (mgr.24367) 87 : cluster [DBG] pgmap v58: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:41.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:41 smithi169 bash[25083]: cluster 2023-12-16T07:37:40.217495+0000 mgr.y (mgr.24367) 87 : cluster [DBG] pgmap v58: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:42.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:37:42 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:37:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:37:43.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:43 smithi007 bash[30436]: cluster 2023-12-16T07:37:42.217940+0000 mgr.y (mgr.24367) 88 : cluster [DBG] pgmap v59: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:43.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:43 smithi007 bash[25009]: cluster 2023-12-16T07:37:42.217940+0000 mgr.y (mgr.24367) 88 : cluster [DBG] pgmap v59: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:43.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:43 smithi169 bash[25083]: cluster 2023-12-16T07:37:42.217940+0000 mgr.y (mgr.24367) 88 : cluster [DBG] pgmap v59: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:43.816 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:37:43 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:37:43] "GET /metrics HTTP/1.1" 200 191488 "" "Prometheus/2.33.4" 2023-12-16T07:37:44.227 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T07:37:44.239 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2023-12-16T07:37:44.240 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2023-12-16T07:37:44.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:44 smithi007 bash[30436]: audit 2023-12-16T07:37:43.437502+0000 mon.a (mon.0) 691 : audit [INF] from='client.? ' entity='client.admin' 2023-12-16T07:37:44.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:44 smithi007 bash[30436]: audit 2023-12-16T07:37:43.453095+0000 mon.a (mon.0) 692 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:37:44.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:44 smithi007 bash[30436]: audit 2023-12-16T07:37:43.454409+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:37:44.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:44 smithi007 bash[30436]: audit 2023-12-16T07:37:43.461273+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:44.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:44 smithi007 bash[25009]: audit 2023-12-16T07:37:43.437502+0000 mon.a (mon.0) 691 : audit [INF] from='client.? ' entity='client.admin' 2023-12-16T07:37:44.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:44 smithi007 bash[25009]: audit 2023-12-16T07:37:43.453095+0000 mon.a (mon.0) 692 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:37:44.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:44 smithi007 bash[25009]: audit 2023-12-16T07:37:43.454409+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:37:44.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:44 smithi007 bash[25009]: audit 2023-12-16T07:37:43.461273+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:44 smithi169 bash[25083]: audit 2023-12-16T07:37:43.437502+0000 mon.a (mon.0) 691 : audit [INF] from='client.? ' entity='client.admin' 2023-12-16T07:37:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:44 smithi169 bash[25083]: audit 2023-12-16T07:37:43.453095+0000 mon.a (mon.0) 692 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:37:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:44 smithi169 bash[25083]: audit 2023-12-16T07:37:43.454409+0000 mon.a (mon.0) 693 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:37:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:44 smithi169 bash[25083]: audit 2023-12-16T07:37:43.461273+0000 mon.a (mon.0) 694 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:37:45.789 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:45 smithi007 bash[30436]: cluster 2023-12-16T07:37:44.218499+0000 mgr.y (mgr.24367) 89 : cluster [DBG] pgmap v60: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:45.790 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:45 smithi007 bash[25009]: cluster 2023-12-16T07:37:44.218499+0000 mgr.y (mgr.24367) 89 : cluster [DBG] pgmap v60: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:45 smithi169 bash[25083]: cluster 2023-12-16T07:37:44.218499+0000 mgr.y (mgr.24367) 89 : cluster [DBG] pgmap v60: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:47.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:47 smithi169 bash[25083]: cluster 2023-12-16T07:37:46.219127+0000 mgr.y (mgr.24367) 90 : cluster [DBG] pgmap v61: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:48.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:47 smithi007 bash[25009]: cluster 2023-12-16T07:37:46.219127+0000 mgr.y (mgr.24367) 90 : cluster [DBG] pgmap v61: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:48.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:47 smithi007 bash[30436]: cluster 2023-12-16T07:37:46.219127+0000 mgr.y (mgr.24367) 90 : cluster [DBG] pgmap v61: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:49.176 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:37:49.176 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "32865a06-1e64-4a08-af96-4aecb4d3e91c", 2023-12-16T07:37:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "r", 2023-12-16T07:37:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "current_period": "952eb9e4-41f1-4b05-9021-57da5102c5d4", 2023-12-16T07:37:49.177 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 1 2023-12-16T07:37:49.177 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:37:49.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:49 smithi007 bash[25009]: cluster 2023-12-16T07:37:48.129069+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2023-12-16T07:37:49.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:49 smithi007 bash[25009]: audit 2023-12-16T07:37:48.133227+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.7:0/3797332288' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-12-16T07:37:49.397 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:49 smithi007 bash[25009]: audit 2023-12-16T07:37:48.134037+0000 mon.a (mon.0) 696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-12-16T07:37:49.397 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:49 smithi007 bash[25009]: cluster 2023-12-16T07:37:48.219834+0000 mgr.y (mgr.24367) 91 : cluster [DBG] pgmap v63: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:49.397 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:49 smithi007 bash[30436]: cluster 2023-12-16T07:37:48.129069+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2023-12-16T07:37:49.397 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:49 smithi007 bash[30436]: audit 2023-12-16T07:37:48.133227+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.7:0/3797332288' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-12-16T07:37:49.397 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:49 smithi007 bash[30436]: audit 2023-12-16T07:37:48.134037+0000 mon.a (mon.0) 696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-12-16T07:37:49.397 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:49 smithi007 bash[30436]: cluster 2023-12-16T07:37:48.219834+0000 mgr.y (mgr.24367) 91 : cluster [DBG] pgmap v63: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:49.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:49 smithi169 bash[25083]: cluster 2023-12-16T07:37:48.129069+0000 mon.a (mon.0) 695 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2023-12-16T07:37:49.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:49 smithi169 bash[25083]: audit 2023-12-16T07:37:48.133227+0000 mon.c (mon.1) 17 : audit [INF] from='client.? 172.21.15.7:0/3797332288' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-12-16T07:37:49.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:49 smithi169 bash[25083]: audit 2023-12-16T07:37:48.134037+0000 mon.a (mon.0) 696 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-12-16T07:37:49.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:49 smithi169 bash[25083]: cluster 2023-12-16T07:37:48.219834+0000 mgr.y (mgr.24367) 91 : cluster [DBG] pgmap v63: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:49.967 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2023-12-16T07:37:50.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:50 smithi007 bash[30436]: audit 2023-12-16T07:37:49.132123+0000 mon.a (mon.0) 697 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-12-16T07:37:50.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:50 smithi007 bash[30436]: cluster 2023-12-16T07:37:49.133446+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2023-12-16T07:37:50.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:50 smithi007 bash[25009]: audit 2023-12-16T07:37:49.132123+0000 mon.a (mon.0) 697 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-12-16T07:37:50.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:50 smithi007 bash[25009]: cluster 2023-12-16T07:37:49.133446+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2023-12-16T07:37:50.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:50 smithi169 bash[25083]: audit 2023-12-16T07:37:49.132123+0000 mon.a (mon.0) 697 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-12-16T07:37:50.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:50 smithi169 bash[25083]: cluster 2023-12-16T07:37:49.133446+0000 mon.a (mon.0) 698 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2023-12-16T07:37:51.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:51 smithi007 bash[25009]: cluster 2023-12-16T07:37:50.151155+0000 mon.a (mon.0) 699 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2023-12-16T07:37:51.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:51 smithi007 bash[25009]: cluster 2023-12-16T07:37:50.220475+0000 mgr.y (mgr.24367) 92 : cluster [DBG] pgmap v66: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:51.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:51 smithi007 bash[25009]: audit 2023-12-16T07:37:50.526774+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:37:51.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:51 smithi007 bash[25009]: audit 2023-12-16T07:37:50.537730+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:37:51.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:51 smithi007 bash[30436]: cluster 2023-12-16T07:37:50.151155+0000 mon.a (mon.0) 699 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2023-12-16T07:37:51.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:51 smithi007 bash[30436]: cluster 2023-12-16T07:37:50.220475+0000 mgr.y (mgr.24367) 92 : cluster [DBG] pgmap v66: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:51.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:51 smithi007 bash[30436]: audit 2023-12-16T07:37:50.526774+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:37:51.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:51 smithi007 bash[30436]: audit 2023-12-16T07:37:50.537730+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:37:51.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:51 smithi169 bash[25083]: cluster 2023-12-16T07:37:50.151155+0000 mon.a (mon.0) 699 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2023-12-16T07:37:51.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:51 smithi169 bash[25083]: cluster 2023-12-16T07:37:50.220475+0000 mgr.y (mgr.24367) 92 : cluster [DBG] pgmap v66: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:37:51.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:51 smithi169 bash[25083]: audit 2023-12-16T07:37:50.526774+0000 mon.a (mon.0) 700 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:37:51.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:51 smithi169 bash[25083]: audit 2023-12-16T07:37:50.537730+0000 mon.a (mon.0) 701 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:37:52.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:37:52 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:37:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:37:52.926 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:37:52.926 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "f387a27f-dbc3-413a-bc92-7e3c8d05031f", 2023-12-16T07:37:52.926 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default", 2023-12-16T07:37:52.926 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "default", 2023-12-16T07:37:52.926 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": "true", 2023-12-16T07:37:52.926 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [], 2023-12-16T07:37:52.927 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2023-12-16T07:37:52.927 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2023-12-16T07:37:52.927 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "", 2023-12-16T07:37:52.927 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [], 2023-12-16T07:37:52.927 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [], 2023-12-16T07:37:52.927 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "", 2023-12-16T07:37:52.927 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "32865a06-1e64-4a08-af96-4aecb4d3e91c", 2023-12-16T07:37:52.927 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2023-12-16T07:37:52.927 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [] 2023-12-16T07:37:52.927 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:37:52.927 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:37:53.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:53 smithi169 bash[25083]: cluster 2023-12-16T07:37:52.221269+0000 mgr.y (mgr.24367) 93 : cluster [DBG] pgmap v67: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2023-12-16T07:37:53.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:53 smithi007 bash[25009]: cluster 2023-12-16T07:37:52.221269+0000 mgr.y (mgr.24367) 93 : cluster [DBG] pgmap v67: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2023-12-16T07:37:53.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:53 smithi007 bash[30436]: cluster 2023-12-16T07:37:52.221269+0000 mgr.y (mgr.24367) 93 : cluster [DBG] pgmap v67: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2023-12-16T07:37:53.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2023-12-16T07:37:54.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:37:53 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:37:53] "GET /metrics HTTP/1.1" 200 192544 "" "Prometheus/2.33.4" 2023-12-16T07:37:55.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:55 smithi007 bash[25009]: cluster 2023-12-16T07:37:54.221937+0000 mgr.y (mgr.24367) 94 : cluster [DBG] pgmap v68: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2023-12-16T07:37:55.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:55 smithi007 bash[30436]: cluster 2023-12-16T07:37:54.221937+0000 mgr.y (mgr.24367) 94 : cluster [DBG] pgmap v68: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2023-12-16T07:37:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:55 smithi169 bash[25083]: cluster 2023-12-16T07:37:54.221937+0000 mgr.y (mgr.24367) 94 : cluster [DBG] pgmap v68: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2023-12-16T07:37:56.737 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:37:56.737 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "66cd2ffc-73d7-48ea-8f40-c0b5a23fc673", 2023-12-16T07:37:56.737 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "z", 2023-12-16T07:37:56.737 INFO:teuthology.orchestra.run.smithi007.stdout: "domain_root": "z.rgw.meta:root", 2023-12-16T07:37:56.737 INFO:teuthology.orchestra.run.smithi007.stdout: "control_pool": "z.rgw.control", 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "gc_pool": "z.rgw.log:gc", 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "lc_pool": "z.rgw.log:lc", 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "log_pool": "z.rgw.log", 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "intent_log_pool": "z.rgw.log:intent", 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "usage_log_pool": "z.rgw.log:usage", 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "roles_pool": "z.rgw.meta:roles", 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "reshard_pool": "z.rgw.log:reshard", 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "user_email_pool": "z.rgw.meta:users.email", 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "otp_pool": "z.rgw.otp", 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "system_key": { 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "access_key": "", 2023-12-16T07:37:56.738 INFO:teuthology.orchestra.run.smithi007.stdout: "secret_key": "" 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_pools": [ 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: { 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "default-placement", 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "index_pool": "z.rgw.buckets.index", 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": { 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD": { 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "data_pool": "z.rgw.buckets.data" 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: "index_type": 0 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:37:56.739 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:37:56.740 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:37:56.740 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "32865a06-1e64-4a08-af96-4aecb4d3e91c", 2023-12-16T07:37:56.740 INFO:teuthology.orchestra.run.smithi007.stdout: "notif_pool": "z.rgw.log:notif" 2023-12-16T07:37:56.740 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:37:57.468 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2023-12-16T07:37:57.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:57 smithi007 bash[30436]: cluster 2023-12-16T07:37:56.222786+0000 mgr.y (mgr.24367) 95 : cluster [DBG] pgmap v69: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2023-12-16T07:37:57.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:57 smithi007 bash[25009]: cluster 2023-12-16T07:37:56.222786+0000 mgr.y (mgr.24367) 95 : cluster [DBG] pgmap v69: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2023-12-16T07:37:57.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:57 smithi169 bash[25083]: cluster 2023-12-16T07:37:56.222786+0000 mgr.y (mgr.24367) 95 : cluster [DBG] pgmap v69: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2023-12-16T07:37:59.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:37:59 smithi007 bash[25009]: cluster 2023-12-16T07:37:58.223387+0000 mgr.y (mgr.24367) 96 : cluster [DBG] pgmap v70: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 450 B/s rd, 1.2 KiB/s wr, 1 op/s 2023-12-16T07:37:59.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:37:59 smithi007 bash[30436]: cluster 2023-12-16T07:37:58.223387+0000 mgr.y (mgr.24367) 96 : cluster [DBG] pgmap v70: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 450 B/s rd, 1.2 KiB/s wr, 1 op/s 2023-12-16T07:37:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:37:59 smithi169 bash[25083]: cluster 2023-12-16T07:37:58.223387+0000 mgr.y (mgr.24367) 96 : cluster [DBG] pgmap v70: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 450 B/s rd, 1.2 KiB/s wr, 1 op/s 2023-12-16T07:38:01.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:01 smithi007 bash[25009]: cluster 2023-12-16T07:38:00.223987+0000 mgr.y (mgr.24367) 97 : cluster [DBG] pgmap v71: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 406 B/s rd, 1.1 KiB/s wr, 1 op/s 2023-12-16T07:38:01.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:01 smithi007 bash[30436]: cluster 2023-12-16T07:38:00.223987+0000 mgr.y (mgr.24367) 97 : cluster [DBG] pgmap v71: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 406 B/s rd, 1.1 KiB/s wr, 1 op/s 2023-12-16T07:38:01.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:01 smithi169 bash[25083]: cluster 2023-12-16T07:38:00.223987+0000 mgr.y (mgr.24367) 97 : cluster [DBG] pgmap v71: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 406 B/s rd, 1.1 KiB/s wr, 1 op/s 2023-12-16T07:38:02.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:02 smithi007 bash[30436]: cluster 2023-12-16T07:38:01.307566+0000 mon.a (mon.0) 702 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2023-12-16T07:38:02.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:02 smithi007 bash[30436]: audit 2023-12-16T07:38:01.309559+0000 mon.c (mon.1) 18 : audit [INF] from='client.? 172.21.15.7:0/1371745387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2023-12-16T07:38:02.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:02 smithi007 bash[30436]: audit 2023-12-16T07:38:01.313543+0000 mon.a (mon.0) 703 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2023-12-16T07:38:02.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:02 smithi007 bash[25009]: cluster 2023-12-16T07:38:01.307566+0000 mon.a (mon.0) 702 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2023-12-16T07:38:02.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:02 smithi007 bash[25009]: audit 2023-12-16T07:38:01.309559+0000 mon.c (mon.1) 18 : audit [INF] from='client.? 172.21.15.7:0/1371745387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2023-12-16T07:38:02.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:02 smithi007 bash[25009]: audit 2023-12-16T07:38:01.313543+0000 mon.a (mon.0) 703 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2023-12-16T07:38:02.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:38:02 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:38:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:38:02.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:02 smithi169 bash[25083]: cluster 2023-12-16T07:38:01.307566+0000 mon.a (mon.0) 702 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2023-12-16T07:38:02.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:02 smithi169 bash[25083]: audit 2023-12-16T07:38:01.309559+0000 mon.c (mon.1) 18 : audit [INF] from='client.? 172.21.15.7:0/1371745387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2023-12-16T07:38:02.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:02 smithi169 bash[25083]: audit 2023-12-16T07:38:01.313543+0000 mon.a (mon.0) 703 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2023-12-16T07:38:03.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:03 smithi169 bash[25083]: cluster 2023-12-16T07:38:02.224678+0000 mgr.y (mgr.24367) 98 : cluster [DBG] pgmap v73: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 818 B/s wr, 3 op/s 2023-12-16T07:38:03.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:03 smithi169 bash[25083]: audit 2023-12-16T07:38:02.315537+0000 mon.a (mon.0) 704 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2023-12-16T07:38:03.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:03 smithi169 bash[25083]: cluster 2023-12-16T07:38:02.315701+0000 mon.a (mon.0) 705 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2023-12-16T07:38:03.716 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:03 smithi007 bash[30436]: cluster 2023-12-16T07:38:02.224678+0000 mgr.y (mgr.24367) 98 : cluster [DBG] pgmap v73: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 818 B/s wr, 3 op/s 2023-12-16T07:38:03.716 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:03 smithi007 bash[30436]: audit 2023-12-16T07:38:02.315537+0000 mon.a (mon.0) 704 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2023-12-16T07:38:03.716 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:03 smithi007 bash[30436]: cluster 2023-12-16T07:38:02.315701+0000 mon.a (mon.0) 705 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2023-12-16T07:38:03.716 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:03 smithi007 bash[25009]: cluster 2023-12-16T07:38:02.224678+0000 mgr.y (mgr.24367) 98 : cluster [DBG] pgmap v73: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 818 B/s wr, 3 op/s 2023-12-16T07:38:03.716 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:03 smithi007 bash[25009]: audit 2023-12-16T07:38:02.315537+0000 mon.a (mon.0) 704 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2023-12-16T07:38:03.716 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:03 smithi007 bash[25009]: cluster 2023-12-16T07:38:02.315701+0000 mon.a (mon.0) 705 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2023-12-16T07:38:04.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:38:03 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:38:03] "GET /metrics HTTP/1.1" 200 192544 "" "Prometheus/2.33.4" 2023-12-16T07:38:04.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:04 smithi169 bash[25083]: cluster 2023-12-16T07:38:03.322290+0000 mon.a (mon.0) 706 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2023-12-16T07:38:04.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:04 smithi169 bash[25083]: audit 2023-12-16T07:38:03.327818+0000 mon.c (mon.1) 19 : audit [INF] from='client.? 172.21.15.7:0/1371745387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2023-12-16T07:38:04.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:04 smithi169 bash[25083]: audit 2023-12-16T07:38:03.336396+0000 mon.a (mon.0) 707 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2023-12-16T07:38:04.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:04 smithi007 bash[30436]: cluster 2023-12-16T07:38:03.322290+0000 mon.a (mon.0) 706 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2023-12-16T07:38:04.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:04 smithi007 bash[30436]: audit 2023-12-16T07:38:03.327818+0000 mon.c (mon.1) 19 : audit [INF] from='client.? 172.21.15.7:0/1371745387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2023-12-16T07:38:04.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:04 smithi007 bash[30436]: audit 2023-12-16T07:38:03.336396+0000 mon.a (mon.0) 707 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2023-12-16T07:38:04.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:04 smithi007 bash[25009]: cluster 2023-12-16T07:38:03.322290+0000 mon.a (mon.0) 706 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2023-12-16T07:38:04.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:04 smithi007 bash[25009]: audit 2023-12-16T07:38:03.327818+0000 mon.c (mon.1) 19 : audit [INF] from='client.? 172.21.15.7:0/1371745387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2023-12-16T07:38:04.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:04 smithi007 bash[25009]: audit 2023-12-16T07:38:03.336396+0000 mon.a (mon.0) 707 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2023-12-16T07:38:05.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:05 smithi169 bash[25083]: cluster 2023-12-16T07:38:04.225397+0000 mgr.y (mgr.24367) 99 : cluster [DBG] pgmap v76: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2023-12-16T07:38:05.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:05 smithi169 bash[25083]: audit 2023-12-16T07:38:04.328111+0000 mon.a (mon.0) 708 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2023-12-16T07:38:05.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:05 smithi169 bash[25083]: cluster 2023-12-16T07:38:04.328248+0000 mon.a (mon.0) 709 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2023-12-16T07:38:05.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:05 smithi007 bash[25009]: cluster 2023-12-16T07:38:04.225397+0000 mgr.y (mgr.24367) 99 : cluster [DBG] pgmap v76: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2023-12-16T07:38:05.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:05 smithi007 bash[25009]: audit 2023-12-16T07:38:04.328111+0000 mon.a (mon.0) 708 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2023-12-16T07:38:05.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:05 smithi007 bash[25009]: cluster 2023-12-16T07:38:04.328248+0000 mon.a (mon.0) 709 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2023-12-16T07:38:05.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:05 smithi007 bash[30436]: cluster 2023-12-16T07:38:04.225397+0000 mgr.y (mgr.24367) 99 : cluster [DBG] pgmap v76: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2023-12-16T07:38:05.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:05 smithi007 bash[30436]: audit 2023-12-16T07:38:04.328111+0000 mon.a (mon.0) 708 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2023-12-16T07:38:05.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:05 smithi007 bash[30436]: cluster 2023-12-16T07:38:04.328248+0000 mon.a (mon.0) 709 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2023-12-16T07:38:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:06 smithi169 bash[25083]: cluster 2023-12-16T07:38:05.344267+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2023-12-16T07:38:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:06 smithi169 bash[25083]: audit 2023-12-16T07:38:05.349297+0000 mon.c (mon.1) 20 : audit [INF] from='client.? 172.21.15.7:0/1461529534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-12-16T07:38:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:06 smithi169 bash[25083]: audit 2023-12-16T07:38:05.352162+0000 mon.a (mon.0) 711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-12-16T07:38:06.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:06 smithi007 bash[25009]: cluster 2023-12-16T07:38:05.344267+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2023-12-16T07:38:06.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:06 smithi007 bash[25009]: audit 2023-12-16T07:38:05.349297+0000 mon.c (mon.1) 20 : audit [INF] from='client.? 172.21.15.7:0/1461529534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-12-16T07:38:06.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:06 smithi007 bash[25009]: audit 2023-12-16T07:38:05.352162+0000 mon.a (mon.0) 711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-12-16T07:38:06.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:06 smithi007 bash[30436]: cluster 2023-12-16T07:38:05.344267+0000 mon.a (mon.0) 710 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2023-12-16T07:38:06.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:06 smithi007 bash[30436]: audit 2023-12-16T07:38:05.349297+0000 mon.c (mon.1) 20 : audit [INF] from='client.? 172.21.15.7:0/1461529534' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-12-16T07:38:06.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:06 smithi007 bash[30436]: audit 2023-12-16T07:38:05.352162+0000 mon.a (mon.0) 711 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2023-12-16T07:38:07.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:07 smithi169 bash[25083]: cluster 2023-12-16T07:38:06.226686+0000 mgr.y (mgr.24367) 100 : cluster [DBG] pgmap v79: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2023-12-16T07:38:07.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:07 smithi169 bash[25083]: audit 2023-12-16T07:38:06.350932+0000 mon.a (mon.0) 712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2023-12-16T07:38:07.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:07 smithi169 bash[25083]: cluster 2023-12-16T07:38:06.351616+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2023-12-16T07:38:07.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:07 smithi169 bash[25083]: audit 2023-12-16T07:38:06.370472+0000 mon.c (mon.1) 21 : audit [INF] from='client.? 172.21.15.7:0/1461529534' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-12-16T07:38:07.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:07 smithi169 bash[25083]: audit 2023-12-16T07:38:06.371338+0000 mon.a (mon.0) 714 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-12-16T07:38:07.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:07 smithi007 bash[25009]: cluster 2023-12-16T07:38:06.226686+0000 mgr.y (mgr.24367) 100 : cluster [DBG] pgmap v79: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2023-12-16T07:38:07.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:07 smithi007 bash[25009]: audit 2023-12-16T07:38:06.350932+0000 mon.a (mon.0) 712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2023-12-16T07:38:07.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:07 smithi007 bash[25009]: cluster 2023-12-16T07:38:06.351616+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2023-12-16T07:38:07.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:07 smithi007 bash[25009]: audit 2023-12-16T07:38:06.370472+0000 mon.c (mon.1) 21 : audit [INF] from='client.? 172.21.15.7:0/1461529534' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-12-16T07:38:07.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:07 smithi007 bash[25009]: audit 2023-12-16T07:38:06.371338+0000 mon.a (mon.0) 714 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-12-16T07:38:07.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:07 smithi007 bash[30436]: cluster 2023-12-16T07:38:06.226686+0000 mgr.y (mgr.24367) 100 : cluster [DBG] pgmap v79: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2023-12-16T07:38:07.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:07 smithi007 bash[30436]: audit 2023-12-16T07:38:06.350932+0000 mon.a (mon.0) 712 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2023-12-16T07:38:07.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:07 smithi007 bash[30436]: cluster 2023-12-16T07:38:06.351616+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2023-12-16T07:38:07.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:07 smithi007 bash[30436]: audit 2023-12-16T07:38:06.370472+0000 mon.c (mon.1) 21 : audit [INF] from='client.? 172.21.15.7:0/1461529534' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-12-16T07:38:07.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:07 smithi007 bash[30436]: audit 2023-12-16T07:38:06.371338+0000 mon.a (mon.0) 714 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-12-16T07:38:08.503 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:38:08.503 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "4fc90568-fe1d-47c6-b0eb-12acb74bfe94", 2023-12-16T07:38:08.504 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 1, 2023-12-16T07:38:08.504 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "952eb9e4-41f1-4b05-9021-57da5102c5d4", 2023-12-16T07:38:08.504 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2023-12-16T07:38:08.504 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2023-12-16T07:38:08.504 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "4fc90568-fe1d-47c6-b0eb-12acb74bfe94", 2023-12-16T07:38:08.504 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2023-12-16T07:38:08.504 INFO:teuthology.orchestra.run.smithi007.stdout: { 2023-12-16T07:38:08.504 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "f387a27f-dbc3-413a-bc92-7e3c8d05031f", 2023-12-16T07:38:08.505 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default", 2023-12-16T07:38:08.505 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "default", 2023-12-16T07:38:08.505 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": "true", 2023-12-16T07:38:08.505 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [], 2023-12-16T07:38:08.505 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2023-12-16T07:38:08.505 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2023-12-16T07:38:08.505 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "66cd2ffc-73d7-48ea-8f40-c0b5a23fc673", 2023-12-16T07:38:08.505 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2023-12-16T07:38:08.505 INFO:teuthology.orchestra.run.smithi007.stdout: { 2023-12-16T07:38:08.505 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "66cd2ffc-73d7-48ea-8f40-c0b5a23fc673", 2023-12-16T07:38:08.505 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "z", 2023-12-16T07:38:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [], 2023-12-16T07:38:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": "false", 2023-12-16T07:38:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": "false", 2023-12-16T07:38:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2023-12-16T07:38:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": "false", 2023-12-16T07:38:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2023-12-16T07:38:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": "true", 2023-12-16T07:38:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2023-12-16T07:38:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "" 2023-12-16T07:38:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:38:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:38:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2023-12-16T07:38:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: { 2023-12-16T07:38:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2023-12-16T07:38:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2023-12-16T07:38:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2023-12-16T07:38:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2023-12-16T07:38:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2023-12-16T07:38:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:38:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:38:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2023-12-16T07:38:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "32865a06-1e64-4a08-af96-4aecb4d3e91c", 2023-12-16T07:38:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2023-12-16T07:38:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [] 2023-12-16T07:38:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:38:08.508 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:38:08.508 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:38:08.508 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2023-12-16T07:38:08.508 INFO:teuthology.orchestra.run.smithi007.stdout: { 2023-12-16T07:38:08.508 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "66cd2ffc-73d7-48ea-8f40-c0b5a23fc673", 2023-12-16T07:38:08.508 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 729708362 2023-12-16T07:38:08.508 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:38:08.508 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2023-12-16T07:38:08.508 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:38:08.509 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "f387a27f-dbc3-413a-bc92-7e3c8d05031f", 2023-12-16T07:38:08.509 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "66cd2ffc-73d7-48ea-8f40-c0b5a23fc673", 2023-12-16T07:38:08.509 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2023-12-16T07:38:08.510 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2023-12-16T07:38:08.510 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2023-12-16T07:38:08.510 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2023-12-16T07:38:08.510 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2023-12-16T07:38:08.510 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2023-12-16T07:38:08.510 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2023-12-16T07:38:08.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:38:08.510 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2023-12-16T07:38:08.510 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2023-12-16T07:38:08.511 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2023-12-16T07:38:08.511 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2023-12-16T07:38:08.511 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2023-12-16T07:38:08.511 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2023-12-16T07:38:08.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:38:08.512 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2023-12-16T07:38:08.512 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2023-12-16T07:38:08.512 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2023-12-16T07:38:08.512 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2023-12-16T07:38:08.512 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2023-12-16T07:38:08.512 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2023-12-16T07:38:08.512 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:38:08.513 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2023-12-16T07:38:08.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2023-12-16T07:38:08.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2023-12-16T07:38:08.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2023-12-16T07:38:08.513 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2023-12-16T07:38:08.513 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2023-12-16T07:38:08.513 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:38:08.514 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2023-12-16T07:38:08.514 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2023-12-16T07:38:08.514 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2023-12-16T07:38:08.514 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2023-12-16T07:38:08.514 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2023-12-16T07:38:08.514 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2023-12-16T07:38:08.514 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:38:08.514 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:38:08.514 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "32865a06-1e64-4a08-af96-4aecb4d3e91c", 2023-12-16T07:38:08.515 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "r", 2023-12-16T07:38:08.515 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2023-12-16T07:38:08.515 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:38:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:08 smithi169 bash[25083]: audit 2023-12-16T07:38:07.365598+0000 mon.a (mon.0) 715 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-12-16T07:38:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:08 smithi169 bash[25083]: cluster 2023-12-16T07:38:07.366194+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2023-12-16T07:38:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:08 smithi169 bash[25083]: audit 2023-12-16T07:38:07.369143+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.7:0/1461529534' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-12-16T07:38:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:08 smithi169 bash[25083]: audit 2023-12-16T07:38:07.373854+0000 mon.a (mon.0) 717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-12-16T07:38:08.718 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:08 smithi007 bash[30436]: audit 2023-12-16T07:38:07.365598+0000 mon.a (mon.0) 715 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-12-16T07:38:08.718 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:08 smithi007 bash[30436]: cluster 2023-12-16T07:38:07.366194+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2023-12-16T07:38:08.718 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:08 smithi007 bash[30436]: audit 2023-12-16T07:38:07.369143+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.7:0/1461529534' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-12-16T07:38:08.718 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:08 smithi007 bash[30436]: audit 2023-12-16T07:38:07.373854+0000 mon.a (mon.0) 717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-12-16T07:38:08.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:08 smithi007 bash[25009]: audit 2023-12-16T07:38:07.365598+0000 mon.a (mon.0) 715 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-12-16T07:38:08.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:08 smithi007 bash[25009]: cluster 2023-12-16T07:38:07.366194+0000 mon.a (mon.0) 716 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2023-12-16T07:38:08.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:08 smithi007 bash[25009]: audit 2023-12-16T07:38:07.369143+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.7:0/1461529534' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-12-16T07:38:08.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:08 smithi007 bash[25009]: audit 2023-12-16T07:38:07.373854+0000 mon.a (mon.0) 717 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2023-12-16T07:38:09.318 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2023-12-16T07:38:09.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:09 smithi007 bash[25009]: cluster 2023-12-16T07:38:08.227478+0000 mgr.y (mgr.24367) 101 : cluster [DBG] pgmap v82: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2023-12-16T07:38:09.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:09 smithi007 bash[25009]: audit 2023-12-16T07:38:08.383349+0000 mon.a (mon.0) 718 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2023-12-16T07:38:09.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:09 smithi007 bash[25009]: cluster 2023-12-16T07:38:08.383435+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2023-12-16T07:38:09.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:09 smithi007 bash[30436]: cluster 2023-12-16T07:38:08.227478+0000 mgr.y (mgr.24367) 101 : cluster [DBG] pgmap v82: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2023-12-16T07:38:09.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:09 smithi007 bash[30436]: audit 2023-12-16T07:38:08.383349+0000 mon.a (mon.0) 718 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2023-12-16T07:38:09.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:09 smithi007 bash[30436]: cluster 2023-12-16T07:38:08.383435+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2023-12-16T07:38:09.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:09 smithi169 bash[25083]: cluster 2023-12-16T07:38:08.227478+0000 mgr.y (mgr.24367) 101 : cluster [DBG] pgmap v82: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2023-12-16T07:38:09.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:09 smithi169 bash[25083]: audit 2023-12-16T07:38:08.383349+0000 mon.a (mon.0) 718 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2023-12-16T07:38:09.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:09 smithi169 bash[25083]: cluster 2023-12-16T07:38:08.383435+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2023-12-16T07:38:11.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:11 smithi169 bash[25083]: cluster 2023-12-16T07:38:10.228199+0000 mgr.y (mgr.24367) 102 : cluster [DBG] pgmap v84: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 209 B/s rd, 419 B/s wr, 1 op/s 2023-12-16T07:38:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:11 smithi007 bash[25009]: cluster 2023-12-16T07:38:10.228199+0000 mgr.y (mgr.24367) 102 : cluster [DBG] pgmap v84: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 209 B/s rd, 419 B/s wr, 1 op/s 2023-12-16T07:38:11.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:11 smithi007 bash[30436]: cluster 2023-12-16T07:38:10.228199+0000 mgr.y (mgr.24367) 102 : cluster [DBG] pgmap v84: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 209 B/s rd, 419 B/s wr, 1 op/s 2023-12-16T07:38:12.625 INFO:teuthology.orchestra.run.smithi007.stdout:Scheduled rgw.foo update... 2023-12-16T07:38:12.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:38:12 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:38:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:38:13.891 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:38:13] "GET /metrics HTTP/1.1" 200 195382 "" "Prometheus/2.33.4" 2023-12-16T07:38:13.891 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: cluster 2023-12-16T07:38:12.229788+0000 mgr.y (mgr.24367) 103 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: audit 2023-12-16T07:38:12.614012+0000 mgr.y (mgr.24367) 104 : audit [DBG] from='client.24625 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: cephadm 2023-12-16T07:38:12.615931+0000 mgr.y (mgr.24367) 105 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: audit 2023-12-16T07:38:12.623120+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: audit 2023-12-16T07:38:12.653996+0000 mon.a (mon.0) 721 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: audit 2023-12-16T07:38:12.655818+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: audit 2023-12-16T07:38:12.921131+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: audit 2023-12-16T07:38:12.933155+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: audit 2023-12-16T07:38:12.941162+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: cephadm 2023-12-16T07:38:12.943104+0000 mgr.y (mgr.24367) 106 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: audit 2023-12-16T07:38:12.959014+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: audit 2023-12-16T07:38:12.960381+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.mglnhl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: audit 2023-12-16T07:38:12.968261+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.mglnhl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: audit 2023-12-16T07:38:12.976776+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: audit 2023-12-16T07:38:12.978003+0000 mon.a (mon.0) 730 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:13.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[30436]: cephadm 2023-12-16T07:38:12.979344+0000 mgr.y (mgr.24367) 107 : cephadm [INF] Deploying daemon rgw.foo.smithi007.mglnhl on smithi007 2023-12-16T07:38:13.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: cluster 2023-12-16T07:38:12.229788+0000 mgr.y (mgr.24367) 103 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-12-16T07:38:13.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: audit 2023-12-16T07:38:12.614012+0000 mgr.y (mgr.24367) 104 : audit [DBG] from='client.24625 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:38:13.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: cephadm 2023-12-16T07:38:12.615931+0000 mgr.y (mgr.24367) 105 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-12-16T07:38:13.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: audit 2023-12-16T07:38:12.623120+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: audit 2023-12-16T07:38:12.653996+0000 mon.a (mon.0) 721 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:13.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: audit 2023-12-16T07:38:12.655818+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:13.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: audit 2023-12-16T07:38:12.921131+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: audit 2023-12-16T07:38:12.933155+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: audit 2023-12-16T07:38:12.941162+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: cephadm 2023-12-16T07:38:12.943104+0000 mgr.y (mgr.24367) 106 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-12-16T07:38:13.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: audit 2023-12-16T07:38:12.959014+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: audit 2023-12-16T07:38:12.960381+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.mglnhl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:38:13.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: audit 2023-12-16T07:38:12.968261+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.mglnhl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-12-16T07:38:13.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: audit 2023-12-16T07:38:12.976776+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: audit 2023-12-16T07:38:12.978003+0000 mon.a (mon.0) 730 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:13.894 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:13 smithi007 bash[25009]: cephadm 2023-12-16T07:38:12.979344+0000 mgr.y (mgr.24367) 107 : cephadm [INF] Deploying daemon rgw.foo.smithi007.mglnhl on smithi007 2023-12-16T07:38:13.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph orch apply rgw smpl' 2023-12-16T07:38:13.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: cluster 2023-12-16T07:38:12.229788+0000 mgr.y (mgr.24367) 103 : cluster [DBG] pgmap v85: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2023-12-16T07:38:13.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: audit 2023-12-16T07:38:12.614012+0000 mgr.y (mgr.24367) 104 : audit [DBG] from='client.24625 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:38:13.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: cephadm 2023-12-16T07:38:12.615931+0000 mgr.y (mgr.24367) 105 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-12-16T07:38:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: audit 2023-12-16T07:38:12.623120+0000 mon.a (mon.0) 720 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: audit 2023-12-16T07:38:12.653996+0000 mon.a (mon.0) 721 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: audit 2023-12-16T07:38:12.655818+0000 mon.a (mon.0) 722 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: audit 2023-12-16T07:38:12.921131+0000 mon.a (mon.0) 723 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: audit 2023-12-16T07:38:12.933155+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: audit 2023-12-16T07:38:12.941162+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: cephadm 2023-12-16T07:38:12.943104+0000 mgr.y (mgr.24367) 106 : cephadm [INF] Saving service rgw.foo spec with placement count:2 2023-12-16T07:38:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: audit 2023-12-16T07:38:12.959014+0000 mon.a (mon.0) 726 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: audit 2023-12-16T07:38:12.960381+0000 mon.a (mon.0) 727 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.mglnhl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:38:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: audit 2023-12-16T07:38:12.968261+0000 mon.a (mon.0) 728 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.mglnhl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-12-16T07:38:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: audit 2023-12-16T07:38:12.976776+0000 mon.a (mon.0) 729 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: audit 2023-12-16T07:38:12.978003+0000 mon.a (mon.0) 730 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:13 smithi169 bash[25083]: cephadm 2023-12-16T07:38:12.979344+0000 mgr.y (mgr.24367) 107 : cephadm [INF] Deploying daemon rgw.foo.smithi007.mglnhl on smithi007 2023-12-16T07:38:15.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:15 smithi007 bash[25009]: cluster 2023-12-16T07:38:14.230540+0000 mgr.y (mgr.24367) 108 : cluster [DBG] pgmap v86: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 909 B/s wr, 2 op/s 2023-12-16T07:38:15.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:15 smithi007 bash[30436]: cluster 2023-12-16T07:38:14.230540+0000 mgr.y (mgr.24367) 108 : cluster [DBG] pgmap v86: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 909 B/s wr, 2 op/s 2023-12-16T07:38:15.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:15 smithi169 bash[25083]: cluster 2023-12-16T07:38:14.230540+0000 mgr.y (mgr.24367) 108 : cluster [DBG] pgmap v86: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 909 B/s wr, 2 op/s 2023-12-16T07:38:18.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:18 smithi007 bash[25009]: cluster 2023-12-16T07:38:16.232093+0000 mgr.y (mgr.24367) 109 : cluster [DBG] pgmap v87: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 808 B/s rd, 808 B/s wr, 2 op/s 2023-12-16T07:38:18.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:18 smithi007 bash[25009]: audit 2023-12-16T07:38:17.093995+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:18.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:18 smithi007 bash[25009]: audit 2023-12-16T07:38:17.095783+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi169.okfkhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:38:18.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:18 smithi007 bash[25009]: audit 2023-12-16T07:38:17.101818+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi169.okfkhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-12-16T07:38:18.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:18 smithi007 bash[25009]: audit 2023-12-16T07:38:17.107987+0000 mon.a (mon.0) 734 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:18.417 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:18 smithi007 bash[25009]: audit 2023-12-16T07:38:17.109335+0000 mon.a (mon.0) 735 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:18.417 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:18 smithi007 bash[30436]: cluster 2023-12-16T07:38:16.232093+0000 mgr.y (mgr.24367) 109 : cluster [DBG] pgmap v87: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 808 B/s rd, 808 B/s wr, 2 op/s 2023-12-16T07:38:18.417 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:18 smithi007 bash[30436]: audit 2023-12-16T07:38:17.093995+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:18.417 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:18 smithi007 bash[30436]: audit 2023-12-16T07:38:17.095783+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi169.okfkhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:38:18.417 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:18 smithi007 bash[30436]: audit 2023-12-16T07:38:17.101818+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi169.okfkhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-12-16T07:38:18.417 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:18 smithi007 bash[30436]: audit 2023-12-16T07:38:17.107987+0000 mon.a (mon.0) 734 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:18.417 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:18 smithi007 bash[30436]: audit 2023-12-16T07:38:17.109335+0000 mon.a (mon.0) 735 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:18.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:18 smithi169 bash[25083]: cluster 2023-12-16T07:38:16.232093+0000 mgr.y (mgr.24367) 109 : cluster [DBG] pgmap v87: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 808 B/s rd, 808 B/s wr, 2 op/s 2023-12-16T07:38:18.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:18 smithi169 bash[25083]: audit 2023-12-16T07:38:17.093995+0000 mon.a (mon.0) 731 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:18.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:18 smithi169 bash[25083]: audit 2023-12-16T07:38:17.095783+0000 mon.a (mon.0) 732 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi169.okfkhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:38:18.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:18 smithi169 bash[25083]: audit 2023-12-16T07:38:17.101818+0000 mon.a (mon.0) 733 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi169.okfkhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-12-16T07:38:18.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:18 smithi169 bash[25083]: audit 2023-12-16T07:38:17.107987+0000 mon.a (mon.0) 734 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:18.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:18 smithi169 bash[25083]: audit 2023-12-16T07:38:17.109335+0000 mon.a (mon.0) 735 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:18.899 INFO:teuthology.orchestra.run.smithi007.stdout:Scheduled rgw.smpl update... 2023-12-16T07:38:19.172 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:19 smithi169 bash[25083]: cephadm 2023-12-16T07:38:17.110375+0000 mgr.y (mgr.24367) 110 : cephadm [INF] Deploying daemon rgw.foo.smithi169.okfkhg on smithi169 2023-12-16T07:38:19.172 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:19 smithi169 bash[25083]: audit 2023-12-16T07:38:18.896966+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:19.479 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:19 smithi007 bash[30436]: cephadm 2023-12-16T07:38:17.110375+0000 mgr.y (mgr.24367) 110 : cephadm [INF] Deploying daemon rgw.foo.smithi169.okfkhg on smithi169 2023-12-16T07:38:19.480 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:19 smithi007 bash[30436]: audit 2023-12-16T07:38:18.896966+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:19.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:19 smithi007 bash[25009]: cephadm 2023-12-16T07:38:17.110375+0000 mgr.y (mgr.24367) 110 : cephadm [INF] Deploying daemon rgw.foo.smithi169.okfkhg on smithi169 2023-12-16T07:38:19.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:19 smithi007 bash[25009]: audit 2023-12-16T07:38:18.896966+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:19.767 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph osd pool create foo' 2023-12-16T07:38:20.405 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:20 smithi169 bash[25083]: cluster 2023-12-16T07:38:18.232853+0000 mgr.y (mgr.24367) 111 : cluster [DBG] pgmap v88: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2023-12-16T07:38:20.405 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:20 smithi169 bash[25083]: audit 2023-12-16T07:38:18.891121+0000 mgr.y (mgr.24367) 112 : audit [DBG] from='client.24640 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:38:20.405 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:20 smithi169 bash[25083]: cephadm 2023-12-16T07:38:18.892440+0000 mgr.y (mgr.24367) 113 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-12-16T07:38:20.405 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:20 smithi169 bash[25083]: audit 2023-12-16T07:38:19.451030+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:20.405 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:20 smithi169 bash[25083]: audit 2023-12-16T07:38:19.454143+0000 mon.a (mon.0) 738 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:20.405 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:20 smithi169 bash[25083]: audit 2023-12-16T07:38:19.456123+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:20.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:20 smithi007 bash[25009]: cluster 2023-12-16T07:38:18.232853+0000 mgr.y (mgr.24367) 111 : cluster [DBG] pgmap v88: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2023-12-16T07:38:20.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:20 smithi007 bash[25009]: audit 2023-12-16T07:38:18.891121+0000 mgr.y (mgr.24367) 112 : audit [DBG] from='client.24640 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:38:20.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:20 smithi007 bash[25009]: cephadm 2023-12-16T07:38:18.892440+0000 mgr.y (mgr.24367) 113 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-12-16T07:38:20.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:20 smithi007 bash[25009]: audit 2023-12-16T07:38:19.451030+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:20.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:20 smithi007 bash[25009]: audit 2023-12-16T07:38:19.454143+0000 mon.a (mon.0) 738 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:20.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:20 smithi007 bash[25009]: audit 2023-12-16T07:38:19.456123+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:20.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:20 smithi007 bash[30436]: cluster 2023-12-16T07:38:18.232853+0000 mgr.y (mgr.24367) 111 : cluster [DBG] pgmap v88: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2023-12-16T07:38:20.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:20 smithi007 bash[30436]: audit 2023-12-16T07:38:18.891121+0000 mgr.y (mgr.24367) 112 : audit [DBG] from='client.24640 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:38:20.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:20 smithi007 bash[30436]: cephadm 2023-12-16T07:38:18.892440+0000 mgr.y (mgr.24367) 113 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-12-16T07:38:20.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:20 smithi007 bash[30436]: audit 2023-12-16T07:38:19.451030+0000 mon.a (mon.0) 737 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:20.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:20 smithi007 bash[30436]: audit 2023-12-16T07:38:19.454143+0000 mon.a (mon.0) 738 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:20.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:20 smithi007 bash[30436]: audit 2023-12-16T07:38:19.456123+0000 mon.a (mon.0) 739 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:21.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:21 smithi007 bash[25009]: cluster 2023-12-16T07:38:20.233597+0000 mgr.y (mgr.24367) 114 : cluster [DBG] pgmap v89: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 604 B/s wr, 1 op/s 2023-12-16T07:38:21.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:21 smithi007 bash[25009]: audit 2023-12-16T07:38:20.323849+0000 mon.a (mon.0) 740 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:21.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:21 smithi007 bash[30436]: cluster 2023-12-16T07:38:20.233597+0000 mgr.y (mgr.24367) 114 : cluster [DBG] pgmap v89: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 604 B/s wr, 1 op/s 2023-12-16T07:38:21.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:21 smithi007 bash[30436]: audit 2023-12-16T07:38:20.323849+0000 mon.a (mon.0) 740 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:21 smithi169 bash[25083]: cluster 2023-12-16T07:38:20.233597+0000 mgr.y (mgr.24367) 114 : cluster [DBG] pgmap v89: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 604 B/s wr, 1 op/s 2023-12-16T07:38:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:21 smithi169 bash[25083]: audit 2023-12-16T07:38:20.323849+0000 mon.a (mon.0) 740 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:22.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:38:22 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:38:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:38:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:23 smithi169 bash[25083]: cluster 2023-12-16T07:38:22.235184+0000 mgr.y (mgr.24367) 115 : cluster [DBG] pgmap v90: 129 pgs: 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 3.4 KiB/s wr, 168 op/s 2023-12-16T07:38:23.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:23 smithi007 bash[30436]: cluster 2023-12-16T07:38:22.235184+0000 mgr.y (mgr.24367) 115 : cluster [DBG] pgmap v90: 129 pgs: 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 3.4 KiB/s wr, 168 op/s 2023-12-16T07:38:23.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:23 smithi007 bash[25009]: cluster 2023-12-16T07:38:22.235184+0000 mgr.y (mgr.24367) 115 : cluster [DBG] pgmap v90: 129 pgs: 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s rd, 3.4 KiB/s wr, 168 op/s 2023-12-16T07:38:24.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:38:23 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:38:23] "GET /metrics HTTP/1.1" 200 202541 "" "Prometheus/2.33.4" 2023-12-16T07:38:24.175 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'foo' created 2023-12-16T07:38:24.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:24 smithi169 bash[25083]: audit 2023-12-16T07:38:23.193001+0000 mon.a (mon.0) 741 : audit [INF] from='client.? 172.21.15.7:0/2395474338' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2023-12-16T07:38:24.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:24 smithi169 bash[25083]: audit 2023-12-16T07:38:23.641709+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:24.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:24 smithi169 bash[25083]: audit 2023-12-16T07:38:23.647473+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:24.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:24 smithi169 bash[25083]: cephadm 2023-12-16T07:38:23.926072+0000 mgr.y (mgr.24367) 116 : cephadm [INF] Checking dashboard <-> RGW credentials 2023-12-16T07:38:24.439 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:24 smithi007 bash[25009]: audit 2023-12-16T07:38:23.193001+0000 mon.a (mon.0) 741 : audit [INF] from='client.? 172.21.15.7:0/2395474338' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2023-12-16T07:38:24.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:24 smithi007 bash[25009]: audit 2023-12-16T07:38:23.641709+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:24.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:24 smithi007 bash[25009]: audit 2023-12-16T07:38:23.647473+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:24.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:24 smithi007 bash[25009]: cephadm 2023-12-16T07:38:23.926072+0000 mgr.y (mgr.24367) 116 : cephadm [INF] Checking dashboard <-> RGW credentials 2023-12-16T07:38:24.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:24 smithi007 bash[30436]: audit 2023-12-16T07:38:23.193001+0000 mon.a (mon.0) 741 : audit [INF] from='client.? 172.21.15.7:0/2395474338' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2023-12-16T07:38:24.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:24 smithi007 bash[30436]: audit 2023-12-16T07:38:23.641709+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:24.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:24 smithi007 bash[30436]: audit 2023-12-16T07:38:23.647473+0000 mon.a (mon.0) 743 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:24.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:24 smithi007 bash[30436]: cephadm 2023-12-16T07:38:23.926072+0000 mgr.y (mgr.24367) 116 : cephadm [INF] Checking dashboard <-> RGW credentials 2023-12-16T07:38:25.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[25009]: audit 2023-12-16T07:38:24.161033+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.7:0/2395474338' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2023-12-16T07:38:25.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[25009]: cluster 2023-12-16T07:38:24.161152+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2023-12-16T07:38:25.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[25009]: cluster 2023-12-16T07:38:24.236290+0000 mgr.y (mgr.24367) 117 : cluster [DBG] pgmap v92: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 3.4 KiB/s wr, 200 op/s 2023-12-16T07:38:25.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[25009]: audit 2023-12-16T07:38:24.500343+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[25009]: audit 2023-12-16T07:38:24.515795+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[25009]: audit 2023-12-16T07:38:24.523609+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[25009]: cephadm 2023-12-16T07:38:24.530177+0000 mgr.y (mgr.24367) 118 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-12-16T07:38:25.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[25009]: audit 2023-12-16T07:38:24.535320+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[25009]: audit 2023-12-16T07:38:24.536656+0000 mon.a (mon.0) 750 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.tizuzt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:38:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[25009]: audit 2023-12-16T07:38:24.543818+0000 mon.a (mon.0) 751 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.tizuzt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-12-16T07:38:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[25009]: audit 2023-12-16T07:38:24.549814+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[25009]: audit 2023-12-16T07:38:24.551265+0000 mon.a (mon.0) 753 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[25009]: cephadm 2023-12-16T07:38:24.553040+0000 mgr.y (mgr.24367) 119 : cephadm [INF] Deploying daemon rgw.smpl.smithi007.tizuzt on smithi007 2023-12-16T07:38:25.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[30436]: audit 2023-12-16T07:38:24.161033+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.7:0/2395474338' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2023-12-16T07:38:25.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[30436]: cluster 2023-12-16T07:38:24.161152+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2023-12-16T07:38:25.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[30436]: cluster 2023-12-16T07:38:24.236290+0000 mgr.y (mgr.24367) 117 : cluster [DBG] pgmap v92: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 3.4 KiB/s wr, 200 op/s 2023-12-16T07:38:25.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[30436]: audit 2023-12-16T07:38:24.500343+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[30436]: audit 2023-12-16T07:38:24.515795+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[30436]: audit 2023-12-16T07:38:24.523609+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[30436]: cephadm 2023-12-16T07:38:24.530177+0000 mgr.y (mgr.24367) 118 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-12-16T07:38:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[30436]: audit 2023-12-16T07:38:24.535320+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[30436]: audit 2023-12-16T07:38:24.536656+0000 mon.a (mon.0) 750 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.tizuzt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:38:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[30436]: audit 2023-12-16T07:38:24.543818+0000 mon.a (mon.0) 751 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.tizuzt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-12-16T07:38:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[30436]: audit 2023-12-16T07:38:24.549814+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[30436]: audit 2023-12-16T07:38:24.551265+0000 mon.a (mon.0) 753 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:25 smithi007 bash[30436]: cephadm 2023-12-16T07:38:24.553040+0000 mgr.y (mgr.24367) 119 : cephadm [INF] Deploying daemon rgw.smpl.smithi007.tizuzt on smithi007 2023-12-16T07:38:25.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'rbd pool init foo' 2023-12-16T07:38:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:25 smithi169 bash[25083]: audit 2023-12-16T07:38:24.161033+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.7:0/2395474338' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2023-12-16T07:38:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:25 smithi169 bash[25083]: cluster 2023-12-16T07:38:24.161152+0000 mon.a (mon.0) 745 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2023-12-16T07:38:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:25 smithi169 bash[25083]: cluster 2023-12-16T07:38:24.236290+0000 mgr.y (mgr.24367) 117 : cluster [DBG] pgmap v92: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s rd, 3.4 KiB/s wr, 200 op/s 2023-12-16T07:38:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:25 smithi169 bash[25083]: audit 2023-12-16T07:38:24.500343+0000 mon.a (mon.0) 746 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:25 smithi169 bash[25083]: audit 2023-12-16T07:38:24.515795+0000 mon.a (mon.0) 747 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:25 smithi169 bash[25083]: audit 2023-12-16T07:38:24.523609+0000 mon.a (mon.0) 748 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:25 smithi169 bash[25083]: cephadm 2023-12-16T07:38:24.530177+0000 mgr.y (mgr.24367) 118 : cephadm [INF] Saving service rgw.smpl spec with placement count:2 2023-12-16T07:38:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:25 smithi169 bash[25083]: audit 2023-12-16T07:38:24.535320+0000 mon.a (mon.0) 749 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:25 smithi169 bash[25083]: audit 2023-12-16T07:38:24.536656+0000 mon.a (mon.0) 750 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.tizuzt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:38:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:25 smithi169 bash[25083]: audit 2023-12-16T07:38:24.543818+0000 mon.a (mon.0) 751 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.tizuzt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-12-16T07:38:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:25 smithi169 bash[25083]: audit 2023-12-16T07:38:24.549814+0000 mon.a (mon.0) 752 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:25 smithi169 bash[25083]: audit 2023-12-16T07:38:24.551265+0000 mon.a (mon.0) 753 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:25 smithi169 bash[25083]: cephadm 2023-12-16T07:38:24.553040+0000 mgr.y (mgr.24367) 119 : cephadm [INF] Deploying daemon rgw.smpl.smithi007.tizuzt on smithi007 2023-12-16T07:38:26.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:26 smithi007 bash[30436]: cluster 2023-12-16T07:38:25.165982+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2023-12-16T07:38:26.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:26 smithi007 bash[30436]: cluster 2023-12-16T07:38:26.166832+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2023-12-16T07:38:26.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:26 smithi007 bash[25009]: cluster 2023-12-16T07:38:25.165982+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2023-12-16T07:38:26.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:26 smithi007 bash[25009]: cluster 2023-12-16T07:38:26.166832+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2023-12-16T07:38:26.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:26 smithi169 bash[25083]: cluster 2023-12-16T07:38:25.165982+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2023-12-16T07:38:26.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:26 smithi169 bash[25083]: cluster 2023-12-16T07:38:26.166832+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2023-12-16T07:38:27.409 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:27 smithi007 bash[30436]: cluster 2023-12-16T07:38:26.238198+0000 mgr.y (mgr.24367) 120 : cluster [DBG] pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 6.0 KiB/s wr, 361 op/s 2023-12-16T07:38:27.409 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:27 smithi007 bash[25009]: cluster 2023-12-16T07:38:26.238198+0000 mgr.y (mgr.24367) 120 : cluster [DBG] pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 6.0 KiB/s wr, 361 op/s 2023-12-16T07:38:27.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:27 smithi169 bash[25083]: cluster 2023-12-16T07:38:26.238198+0000 mgr.y (mgr.24367) 120 : cluster [DBG] pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 6.0 KiB/s wr, 361 op/s 2023-12-16T07:38:30.355 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:30 smithi169 bash[25083]: cluster 2023-12-16T07:38:28.239039+0000 mgr.y (mgr.24367) 121 : cluster [DBG] pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 341 B/s wr, 26 op/s 2023-12-16T07:38:30.355 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:30 smithi169 bash[25083]: audit 2023-12-16T07:38:29.081262+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:30.355 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:30 smithi169 bash[25083]: audit 2023-12-16T07:38:29.083256+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi169.aiyfmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:38:30.355 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:30 smithi169 bash[25083]: audit 2023-12-16T07:38:29.088678+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi169.aiyfmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-12-16T07:38:30.355 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:30 smithi169 bash[25083]: audit 2023-12-16T07:38:29.095723+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:30.355 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:30 smithi169 bash[25083]: audit 2023-12-16T07:38:29.096917+0000 mon.a (mon.0) 760 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:30.355 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:30 smithi169 bash[25083]: cephadm 2023-12-16T07:38:29.098148+0000 mgr.y (mgr.24367) 122 : cephadm [INF] Deploying daemon rgw.smpl.smithi169.aiyfmr on smithi169 2023-12-16T07:38:30.383 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:30 smithi007 bash[30436]: cluster 2023-12-16T07:38:28.239039+0000 mgr.y (mgr.24367) 121 : cluster [DBG] pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 341 B/s wr, 26 op/s 2023-12-16T07:38:30.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:30 smithi007 bash[30436]: audit 2023-12-16T07:38:29.081262+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:30.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:30 smithi007 bash[30436]: audit 2023-12-16T07:38:29.083256+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi169.aiyfmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:38:30.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:30 smithi007 bash[30436]: audit 2023-12-16T07:38:29.088678+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi169.aiyfmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-12-16T07:38:30.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:30 smithi007 bash[30436]: audit 2023-12-16T07:38:29.095723+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:30.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:30 smithi007 bash[30436]: audit 2023-12-16T07:38:29.096917+0000 mon.a (mon.0) 760 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:30.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:30 smithi007 bash[30436]: cephadm 2023-12-16T07:38:29.098148+0000 mgr.y (mgr.24367) 122 : cephadm [INF] Deploying daemon rgw.smpl.smithi169.aiyfmr on smithi169 2023-12-16T07:38:30.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:30 smithi007 bash[25009]: cluster 2023-12-16T07:38:28.239039+0000 mgr.y (mgr.24367) 121 : cluster [DBG] pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 341 B/s wr, 26 op/s 2023-12-16T07:38:30.384 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:30 smithi007 bash[25009]: audit 2023-12-16T07:38:29.081262+0000 mon.a (mon.0) 756 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:30.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:30 smithi007 bash[25009]: audit 2023-12-16T07:38:29.083256+0000 mon.a (mon.0) 757 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi169.aiyfmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:38:30.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:30 smithi007 bash[25009]: audit 2023-12-16T07:38:29.088678+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi169.aiyfmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-12-16T07:38:30.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:30 smithi007 bash[25009]: audit 2023-12-16T07:38:29.095723+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:30.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:30 smithi007 bash[25009]: audit 2023-12-16T07:38:29.096917+0000 mon.a (mon.0) 760 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:30.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:30 smithi007 bash[25009]: cephadm 2023-12-16T07:38:29.098148+0000 mgr.y (mgr.24367) 122 : cephadm [INF] Deploying daemon rgw.smpl.smithi169.aiyfmr on smithi169 2023-12-16T07:38:31.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:31 smithi169 bash[25083]: audit 2023-12-16T07:38:30.498221+0000 mon.c (mon.1) 23 : audit [INF] from='client.? 172.21.15.7:0/1158168085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-12-16T07:38:31.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:31 smithi169 bash[25083]: audit 2023-12-16T07:38:30.498856+0000 mon.a (mon.0) 761 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-12-16T07:38:31.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:31 smithi007 bash[25009]: audit 2023-12-16T07:38:30.498221+0000 mon.c (mon.1) 23 : audit [INF] from='client.? 172.21.15.7:0/1158168085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-12-16T07:38:31.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:31 smithi007 bash[25009]: audit 2023-12-16T07:38:30.498856+0000 mon.a (mon.0) 761 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-12-16T07:38:31.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:31 smithi007 bash[30436]: audit 2023-12-16T07:38:30.498221+0000 mon.c (mon.1) 23 : audit [INF] from='client.? 172.21.15.7:0/1158168085' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-12-16T07:38:31.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:31 smithi007 bash[30436]: audit 2023-12-16T07:38:30.498856+0000 mon.a (mon.0) 761 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2023-12-16T07:38:32.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:38:32 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:38:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:38:32.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:32 smithi169 bash[25083]: cluster 2023-12-16T07:38:30.239768+0000 mgr.y (mgr.24367) 123 : cluster [DBG] pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 336 B/s wr, 25 op/s 2023-12-16T07:38:32.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:32 smithi169 bash[25083]: audit 2023-12-16T07:38:31.094671+0000 mon.a (mon.0) 762 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2023-12-16T07:38:32.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:32 smithi169 bash[25083]: cluster 2023-12-16T07:38:31.094809+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2023-12-16T07:38:32.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:32 smithi169 bash[25083]: audit 2023-12-16T07:38:31.732901+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:32.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:32 smithi169 bash[25083]: audit 2023-12-16T07:38:31.735619+0000 mon.a (mon.0) 765 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:32.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:32 smithi169 bash[25083]: audit 2023-12-16T07:38:31.737366+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:32.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:32 smithi007 bash[25009]: cluster 2023-12-16T07:38:30.239768+0000 mgr.y (mgr.24367) 123 : cluster [DBG] pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 336 B/s wr, 25 op/s 2023-12-16T07:38:32.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:32 smithi007 bash[25009]: audit 2023-12-16T07:38:31.094671+0000 mon.a (mon.0) 762 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2023-12-16T07:38:32.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:32 smithi007 bash[25009]: cluster 2023-12-16T07:38:31.094809+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2023-12-16T07:38:32.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:32 smithi007 bash[25009]: audit 2023-12-16T07:38:31.732901+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:32.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:32 smithi007 bash[25009]: audit 2023-12-16T07:38:31.735619+0000 mon.a (mon.0) 765 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:32.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:32 smithi007 bash[25009]: audit 2023-12-16T07:38:31.737366+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:32.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:32 smithi007 bash[30436]: cluster 2023-12-16T07:38:30.239768+0000 mgr.y (mgr.24367) 123 : cluster [DBG] pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 336 B/s wr, 25 op/s 2023-12-16T07:38:32.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:32 smithi007 bash[30436]: audit 2023-12-16T07:38:31.094671+0000 mon.a (mon.0) 762 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2023-12-16T07:38:32.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:32 smithi007 bash[30436]: cluster 2023-12-16T07:38:31.094809+0000 mon.a (mon.0) 763 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2023-12-16T07:38:32.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:32 smithi007 bash[30436]: audit 2023-12-16T07:38:31.732901+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:32.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:32 smithi007 bash[30436]: audit 2023-12-16T07:38:31.735619+0000 mon.a (mon.0) 765 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:32.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:32 smithi007 bash[30436]: audit 2023-12-16T07:38:31.737366+0000 mon.a (mon.0) 766 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:33.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:33 smithi169 bash[25083]: cluster 2023-12-16T07:38:32.097622+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2023-12-16T07:38:33.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:33 smithi169 bash[25083]: cluster 2023-12-16T07:38:32.240599+0000 mgr.y (mgr.24367) 124 : cluster [DBG] pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 505 B/s wr, 160 op/s 2023-12-16T07:38:33.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:33 smithi007 bash[30436]: cluster 2023-12-16T07:38:32.097622+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2023-12-16T07:38:33.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:33 smithi007 bash[30436]: cluster 2023-12-16T07:38:32.240599+0000 mgr.y (mgr.24367) 124 : cluster [DBG] pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 505 B/s wr, 160 op/s 2023-12-16T07:38:33.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:33 smithi007 bash[25009]: cluster 2023-12-16T07:38:32.097622+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2023-12-16T07:38:33.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:33 smithi007 bash[25009]: cluster 2023-12-16T07:38:32.240599+0000 mgr.y (mgr.24367) 124 : cluster [DBG] pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 505 B/s wr, 160 op/s 2023-12-16T07:38:34.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:38:33 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:38:33] "GET /metrics HTTP/1.1" 200 202541 "" "Prometheus/2.33.4" 2023-12-16T07:38:34.137 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph orch apply iscsi foo u p' 2023-12-16T07:38:34.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:34 smithi169 bash[25083]: cluster 2023-12-16T07:38:33.115914+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2023-12-16T07:38:34.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:34 smithi007 bash[25009]: cluster 2023-12-16T07:38:33.115914+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2023-12-16T07:38:34.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:34 smithi007 bash[30436]: cluster 2023-12-16T07:38:33.115914+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2023-12-16T07:38:35.374 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:35 smithi007 bash[25009]: cluster 2023-12-16T07:38:34.241372+0000 mgr.y (mgr.24367) 125 : cluster [DBG] pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 170 B/s wr, 136 op/s 2023-12-16T07:38:35.375 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:35 smithi007 bash[30436]: cluster 2023-12-16T07:38:34.241372+0000 mgr.y (mgr.24367) 125 : cluster [DBG] pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 170 B/s wr, 136 op/s 2023-12-16T07:38:35.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:35 smithi169 bash[25083]: cluster 2023-12-16T07:38:34.241372+0000 mgr.y (mgr.24367) 125 : cluster [DBG] pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 170 B/s wr, 136 op/s 2023-12-16T07:38:36.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[25009]: audit 2023-12-16T07:38:35.353075+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:36.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[25009]: audit 2023-12-16T07:38:35.613174+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:36.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[25009]: audit 2023-12-16T07:38:35.663143+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:36.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[25009]: cephadm 2023-12-16T07:38:35.668716+0000 mgr.y (mgr.24367) 126 : cephadm [INF] Checking dashboard <-> RGW credentials 2023-12-16T07:38:36.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[25009]: audit 2023-12-16T07:38:36.027339+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:36.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[25009]: audit 2023-12-16T07:38:36.043889+0000 mon.a (mon.0) 773 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:36.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[25009]: audit 2023-12-16T07:38:36.045545+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:36.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[25009]: cephadm 2023-12-16T07:38:36.055621+0000 mgr.y (mgr.24367) 127 : cephadm [INF] Checking dashboard <-> RGW credentials 2023-12-16T07:38:36.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[30436]: audit 2023-12-16T07:38:35.353075+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:36.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[30436]: audit 2023-12-16T07:38:35.613174+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:36.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[30436]: audit 2023-12-16T07:38:35.663143+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:36.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[30436]: cephadm 2023-12-16T07:38:35.668716+0000 mgr.y (mgr.24367) 126 : cephadm [INF] Checking dashboard <-> RGW credentials 2023-12-16T07:38:36.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[30436]: audit 2023-12-16T07:38:36.027339+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:36.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[30436]: audit 2023-12-16T07:38:36.043889+0000 mon.a (mon.0) 773 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:36.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[30436]: audit 2023-12-16T07:38:36.045545+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:36.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:36 smithi007 bash[30436]: cephadm 2023-12-16T07:38:36.055621+0000 mgr.y (mgr.24367) 127 : cephadm [INF] Checking dashboard <-> RGW credentials 2023-12-16T07:38:36.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:36 smithi169 bash[25083]: audit 2023-12-16T07:38:35.353075+0000 mon.a (mon.0) 769 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:36.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:36 smithi169 bash[25083]: audit 2023-12-16T07:38:35.613174+0000 mon.a (mon.0) 770 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:36.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:36 smithi169 bash[25083]: audit 2023-12-16T07:38:35.663143+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:36.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:36 smithi169 bash[25083]: cephadm 2023-12-16T07:38:35.668716+0000 mgr.y (mgr.24367) 126 : cephadm [INF] Checking dashboard <-> RGW credentials 2023-12-16T07:38:36.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:36 smithi169 bash[25083]: audit 2023-12-16T07:38:36.027339+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:36.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:36 smithi169 bash[25083]: audit 2023-12-16T07:38:36.043889+0000 mon.a (mon.0) 773 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:36.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:36 smithi169 bash[25083]: audit 2023-12-16T07:38:36.045545+0000 mon.a (mon.0) 774 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:36.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:36 smithi169 bash[25083]: cephadm 2023-12-16T07:38:36.055621+0000 mgr.y (mgr.24367) 127 : cephadm [INF] Checking dashboard <-> RGW credentials 2023-12-16T07:38:37.466 INFO:teuthology.orchestra.run.smithi007.stdout:Scheduled iscsi.foo update... 2023-12-16T07:38:37.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:37 smithi169 bash[25083]: cluster 2023-12-16T07:38:36.243100+0000 mgr.y (mgr.24367) 128 : cluster [DBG] pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 224 KiB/s rd, 682 B/s wr, 343 op/s 2023-12-16T07:38:37.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:37 smithi169 bash[25083]: audit 2023-12-16T07:38:36.419066+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:37.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:37 smithi169 bash[25083]: cluster 2023-12-16T07:38:37.145568+0000 mon.a (mon.0) 776 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2023-12-16T07:38:37.688 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:37 smithi007 bash[25009]: cluster 2023-12-16T07:38:36.243100+0000 mgr.y (mgr.24367) 128 : cluster [DBG] pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 224 KiB/s rd, 682 B/s wr, 343 op/s 2023-12-16T07:38:37.688 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:37 smithi007 bash[25009]: audit 2023-12-16T07:38:36.419066+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:37.688 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:37 smithi007 bash[25009]: cluster 2023-12-16T07:38:37.145568+0000 mon.a (mon.0) 776 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2023-12-16T07:38:37.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:37 smithi007 bash[30436]: cluster 2023-12-16T07:38:36.243100+0000 mgr.y (mgr.24367) 128 : cluster [DBG] pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 224 KiB/s rd, 682 B/s wr, 343 op/s 2023-12-16T07:38:37.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:37 smithi007 bash[30436]: audit 2023-12-16T07:38:36.419066+0000 mon.a (mon.0) 775 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:37.689 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:37 smithi007 bash[30436]: cluster 2023-12-16T07:38:37.145568+0000 mon.a (mon.0) 776 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2023-12-16T07:38:38.781 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[30436]: audit 2023-12-16T07:38:37.455069+0000 mgr.y (mgr.24367) 129 : audit [DBG] from='client.14925 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:38:38.781 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[30436]: cephadm 2023-12-16T07:38:37.459216+0000 mgr.y (mgr.24367) 130 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2023-12-16T07:38:38.781 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[30436]: audit 2023-12-16T07:38:37.464319+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:38.781 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[30436]: audit 2023-12-16T07:38:37.494148+0000 mon.a (mon.0) 778 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:38.782 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[30436]: audit 2023-12-16T07:38:37.496191+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:38.782 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[30436]: audit 2023-12-16T07:38:37.505357+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:38.782 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[30436]: cephadm 2023-12-16T07:38:37.511128+0000 mgr.y (mgr.24367) 131 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2023-12-16T07:38:38.782 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[30436]: audit 2023-12-16T07:38:37.519490+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-12-16T07:38:38.782 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[30436]: audit 2023-12-16T07:38:37.524857+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-12-16T07:38:38.782 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[30436]: audit 2023-12-16T07:38:37.532012+0000 mon.a (mon.0) 783 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:38.782 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[30436]: cephadm 2023-12-16T07:38:37.533589+0000 mgr.y (mgr.24367) 132 : cephadm [INF] Deploying daemon iscsi.foo.smithi007.crrdaq on smithi007 2023-12-16T07:38:38.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[25009]: audit 2023-12-16T07:38:37.455069+0000 mgr.y (mgr.24367) 129 : audit [DBG] from='client.14925 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:38:38.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[25009]: cephadm 2023-12-16T07:38:37.459216+0000 mgr.y (mgr.24367) 130 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2023-12-16T07:38:38.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[25009]: audit 2023-12-16T07:38:37.464319+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:38.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[25009]: audit 2023-12-16T07:38:37.494148+0000 mon.a (mon.0) 778 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:38.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[25009]: audit 2023-12-16T07:38:37.496191+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:38.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[25009]: audit 2023-12-16T07:38:37.505357+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:38.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[25009]: cephadm 2023-12-16T07:38:37.511128+0000 mgr.y (mgr.24367) 131 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2023-12-16T07:38:38.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[25009]: audit 2023-12-16T07:38:37.519490+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-12-16T07:38:38.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[25009]: audit 2023-12-16T07:38:37.524857+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-12-16T07:38:38.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[25009]: audit 2023-12-16T07:38:37.532012+0000 mon.a (mon.0) 783 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:38.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:38 smithi007 bash[25009]: cephadm 2023-12-16T07:38:37.533589+0000 mgr.y (mgr.24367) 132 : cephadm [INF] Deploying daemon iscsi.foo.smithi007.crrdaq on smithi007 2023-12-16T07:38:38.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'sleep 120' 2023-12-16T07:38:38.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:38 smithi169 bash[25083]: audit 2023-12-16T07:38:37.455069+0000 mgr.y (mgr.24367) 129 : audit [DBG] from='client.14925 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:38:38.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:38 smithi169 bash[25083]: cephadm 2023-12-16T07:38:37.459216+0000 mgr.y (mgr.24367) 130 : cephadm [INF] Saving service iscsi.foo spec with placement count:1 2023-12-16T07:38:38.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:38 smithi169 bash[25083]: audit 2023-12-16T07:38:37.464319+0000 mon.a (mon.0) 777 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:38.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:38 smithi169 bash[25083]: audit 2023-12-16T07:38:37.494148+0000 mon.a (mon.0) 778 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:38.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:38 smithi169 bash[25083]: audit 2023-12-16T07:38:37.496191+0000 mon.a (mon.0) 779 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:38.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:38 smithi169 bash[25083]: audit 2023-12-16T07:38:37.505357+0000 mon.a (mon.0) 780 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:38.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:38 smithi169 bash[25083]: cephadm 2023-12-16T07:38:37.511128+0000 mgr.y (mgr.24367) 131 : cephadm [INF] Checking pool "foo" exists for service iscsi.foo 2023-12-16T07:38:38.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:38 smithi169 bash[25083]: audit 2023-12-16T07:38:37.519490+0000 mon.a (mon.0) 781 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-12-16T07:38:38.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:38 smithi169 bash[25083]: audit 2023-12-16T07:38:37.524857+0000 mon.a (mon.0) 782 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-12-16T07:38:38.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:38 smithi169 bash[25083]: audit 2023-12-16T07:38:37.532012+0000 mon.a (mon.0) 783 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:38.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:38 smithi169 bash[25083]: cephadm 2023-12-16T07:38:37.533589+0000 mgr.y (mgr.24367) 132 : cephadm [INF] Deploying daemon iscsi.foo.smithi007.crrdaq on smithi007 2023-12-16T07:38:39.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:39 smithi007 bash[30436]: cluster 2023-12-16T07:38:38.244039+0000 mgr.y (mgr.24367) 133 : cluster [DBG] pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 499 B/s wr, 201 op/s 2023-12-16T07:38:39.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:39 smithi007 bash[30436]: cluster 2023-12-16T07:38:38.542799+0000 mon.a (mon.0) 784 : cluster [DBG] mgrmap e21: y(active, since 2m), standbys: x 2023-12-16T07:38:39.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:39 smithi007 bash[25009]: cluster 2023-12-16T07:38:38.244039+0000 mgr.y (mgr.24367) 133 : cluster [DBG] pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 499 B/s wr, 201 op/s 2023-12-16T07:38:39.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:39 smithi007 bash[25009]: cluster 2023-12-16T07:38:38.542799+0000 mon.a (mon.0) 784 : cluster [DBG] mgrmap e21: y(active, since 2m), standbys: x 2023-12-16T07:38:39.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:39 smithi169 bash[25083]: cluster 2023-12-16T07:38:38.244039+0000 mgr.y (mgr.24367) 133 : cluster [DBG] pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 131 KiB/s rd, 499 B/s wr, 201 op/s 2023-12-16T07:38:39.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:39 smithi169 bash[25083]: cluster 2023-12-16T07:38:38.542799+0000 mon.a (mon.0) 784 : cluster [DBG] mgrmap e21: y(active, since 2m), standbys: x 2023-12-16T07:38:41.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:41 smithi007 bash[30436]: cluster 2023-12-16T07:38:40.244843+0000 mgr.y (mgr.24367) 134 : cluster [DBG] pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 383 B/s wr, 155 op/s 2023-12-16T07:38:41.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:41 smithi007 bash[30436]: audit 2023-12-16T07:38:41.402841+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:41.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:41 smithi007 bash[30436]: audit 2023-12-16T07:38:41.404842+0000 mon.a (mon.0) 786 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:41.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:41 smithi007 bash[30436]: audit 2023-12-16T07:38:41.405901+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:41.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:41 smithi007 bash[25009]: cluster 2023-12-16T07:38:40.244843+0000 mgr.y (mgr.24367) 134 : cluster [DBG] pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 383 B/s wr, 155 op/s 2023-12-16T07:38:41.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:41 smithi007 bash[25009]: audit 2023-12-16T07:38:41.402841+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:41.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:41 smithi007 bash[25009]: audit 2023-12-16T07:38:41.404842+0000 mon.a (mon.0) 786 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:41.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:41 smithi007 bash[25009]: audit 2023-12-16T07:38:41.405901+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:41.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:41 smithi169 bash[25083]: cluster 2023-12-16T07:38:40.244843+0000 mgr.y (mgr.24367) 134 : cluster [DBG] pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 383 B/s wr, 155 op/s 2023-12-16T07:38:41.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:41 smithi169 bash[25083]: audit 2023-12-16T07:38:41.402841+0000 mon.a (mon.0) 785 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:41.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:41 smithi169 bash[25083]: audit 2023-12-16T07:38:41.404842+0000 mon.a (mon.0) 786 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:41.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:41 smithi169 bash[25083]: audit 2023-12-16T07:38:41.405901+0000 mon.a (mon.0) 787 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:42.684 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:38:42 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:38:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:38:43.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:43 smithi169 bash[25083]: cluster 2023-12-16T07:38:42.246634+0000 mgr.y (mgr.24367) 135 : cluster [DBG] pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 336 B/s wr, 138 op/s 2023-12-16T07:38:43.456 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:43 smithi007 bash[25009]: cluster 2023-12-16T07:38:42.246634+0000 mgr.y (mgr.24367) 135 : cluster [DBG] pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 336 B/s wr, 138 op/s 2023-12-16T07:38:43.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:43 smithi007 bash[30436]: cluster 2023-12-16T07:38:42.246634+0000 mgr.y (mgr.24367) 135 : cluster [DBG] pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 336 B/s wr, 138 op/s 2023-12-16T07:38:43.816 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:38:43 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:38:43] "GET /metrics HTTP/1.1" 200 213390 "" "Prometheus/2.33.4" 2023-12-16T07:38:43.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:38:43 smithi007 bash[49264]: level=warn ts=2023-12-16T07:38:43.533Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:38:43.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:38:43 smithi007 bash[49264]: level=warn ts=2023-12-16T07:38:43.534Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2023-12-16T07:38:44.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:44 smithi007 bash[25009]: audit 2023-12-16T07:38:43.920026+0000 mon.a (mon.0) 788 : audit [DBG] from='client.? 172.21.15.7:0/1813018608' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-12-16T07:38:44.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:44 smithi007 bash[30436]: audit 2023-12-16T07:38:43.920026+0000 mon.a (mon.0) 788 : audit [DBG] from='client.? 172.21.15.7:0/1813018608' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-12-16T07:38:44.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:44 smithi169 bash[25083]: audit 2023-12-16T07:38:43.920026+0000 mon.a (mon.0) 788 : audit [DBG] from='client.? 172.21.15.7:0/1813018608' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-12-16T07:38:45.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:45 smithi169 bash[25083]: cluster 2023-12-16T07:38:44.247189+0000 mgr.y (mgr.24367) 136 : cluster [DBG] pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 307 B/s wr, 126 op/s 2023-12-16T07:38:45.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:45 smithi169 bash[25083]: audit 2023-12-16T07:38:44.297539+0000 mon.c (mon.1) 24 : audit [INF] from='client.? 172.21.15.7:0/1456008292' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3947556342"}]: dispatch 2023-12-16T07:38:45.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:45 smithi169 bash[25083]: audit 2023-12-16T07:38:44.298557+0000 mon.a (mon.0) 789 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3947556342"}]: dispatch 2023-12-16T07:38:45.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:45 smithi007 bash[30436]: cluster 2023-12-16T07:38:44.247189+0000 mgr.y (mgr.24367) 136 : cluster [DBG] pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 307 B/s wr, 126 op/s 2023-12-16T07:38:45.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:45 smithi007 bash[30436]: audit 2023-12-16T07:38:44.297539+0000 mon.c (mon.1) 24 : audit [INF] from='client.? 172.21.15.7:0/1456008292' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3947556342"}]: dispatch 2023-12-16T07:38:45.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:45 smithi007 bash[30436]: audit 2023-12-16T07:38:44.298557+0000 mon.a (mon.0) 789 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3947556342"}]: dispatch 2023-12-16T07:38:45.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:45 smithi007 bash[25009]: cluster 2023-12-16T07:38:44.247189+0000 mgr.y (mgr.24367) 136 : cluster [DBG] pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 307 B/s wr, 126 op/s 2023-12-16T07:38:45.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:45 smithi007 bash[25009]: audit 2023-12-16T07:38:44.297539+0000 mon.c (mon.1) 24 : audit [INF] from='client.? 172.21.15.7:0/1456008292' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3947556342"}]: dispatch 2023-12-16T07:38:45.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:45 smithi007 bash[25009]: audit 2023-12-16T07:38:44.298557+0000 mon.a (mon.0) 789 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3947556342"}]: dispatch 2023-12-16T07:38:46.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:46 smithi169 bash[25083]: audit 2023-12-16T07:38:45.168865+0000 mon.a (mon.0) 790 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3947556342"}]': finished 2023-12-16T07:38:46.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:46 smithi169 bash[25083]: cluster 2023-12-16T07:38:45.168975+0000 mon.a (mon.0) 791 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2023-12-16T07:38:46.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:46 smithi169 bash[25083]: audit 2023-12-16T07:38:45.359744+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:46.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:46 smithi169 bash[25083]: audit 2023-12-16T07:38:45.593922+0000 mon.c (mon.1) 25 : audit [INF] from='client.? 172.21.15.7:0/1020624530' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3947556342"}]: dispatch 2023-12-16T07:38:46.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:46 smithi169 bash[25083]: audit 2023-12-16T07:38:45.595001+0000 mon.a (mon.0) 793 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3947556342"}]: dispatch 2023-12-16T07:38:46.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:46 smithi007 bash[30436]: audit 2023-12-16T07:38:45.168865+0000 mon.a (mon.0) 790 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3947556342"}]': finished 2023-12-16T07:38:46.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:46 smithi007 bash[30436]: cluster 2023-12-16T07:38:45.168975+0000 mon.a (mon.0) 791 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2023-12-16T07:38:46.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:46 smithi007 bash[30436]: audit 2023-12-16T07:38:45.359744+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:46.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:46 smithi007 bash[30436]: audit 2023-12-16T07:38:45.593922+0000 mon.c (mon.1) 25 : audit [INF] from='client.? 172.21.15.7:0/1020624530' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3947556342"}]: dispatch 2023-12-16T07:38:46.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:46 smithi007 bash[30436]: audit 2023-12-16T07:38:45.595001+0000 mon.a (mon.0) 793 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3947556342"}]: dispatch 2023-12-16T07:38:46.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:46 smithi007 bash[25009]: audit 2023-12-16T07:38:45.168865+0000 mon.a (mon.0) 790 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3947556342"}]': finished 2023-12-16T07:38:46.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:46 smithi007 bash[25009]: cluster 2023-12-16T07:38:45.168975+0000 mon.a (mon.0) 791 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2023-12-16T07:38:46.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:46 smithi007 bash[25009]: audit 2023-12-16T07:38:45.359744+0000 mon.a (mon.0) 792 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:46.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:46 smithi007 bash[25009]: audit 2023-12-16T07:38:45.593922+0000 mon.c (mon.1) 25 : audit [INF] from='client.? 172.21.15.7:0/1020624530' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3947556342"}]: dispatch 2023-12-16T07:38:46.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:46 smithi007 bash[25009]: audit 2023-12-16T07:38:45.595001+0000 mon.a (mon.0) 793 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3947556342"}]: dispatch 2023-12-16T07:38:47.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: cluster 2023-12-16T07:38:46.248826+0000 mgr.y (mgr.24367) 137 : cluster [DBG] pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 112 B/s wr, 3 op/s 2023-12-16T07:38:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.371900+0000 mon.a (mon.0) 794 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3947556342"}]': finished 2023-12-16T07:38:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: cluster 2023-12-16T07:38:46.372042+0000 mon.a (mon.0) 795 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2023-12-16T07:38:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.412295+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.417931+0000 mon.a (mon.0) 797 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.428295+0000 mon.a (mon.0) 798 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:38:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.429281+0000 mgr.y (mgr.24367) 138 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:38:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: cephadm 2023-12-16T07:38:46.430976+0000 mgr.y (mgr.24367) 139 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2023-12-16T07:38:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.431293+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:38:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.431732+0000 mgr.y (mgr.24367) 140 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:38:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.438490+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.443436+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:38:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.444182+0000 mgr.y (mgr.24367) 141 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:38:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.458187+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.460560+0000 mon.a (mon.0) 803 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:47.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.461556+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:47.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.469866+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.738871+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.7:0/278578181' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3815256908"}]: dispatch 2023-12-16T07:38:47.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:47 smithi169 bash[25083]: audit 2023-12-16T07:38:46.739643+0000 mon.a (mon.0) 806 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3815256908"}]: dispatch 2023-12-16T07:38:47.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: cluster 2023-12-16T07:38:46.248826+0000 mgr.y (mgr.24367) 137 : cluster [DBG] pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 112 B/s wr, 3 op/s 2023-12-16T07:38:47.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.371900+0000 mon.a (mon.0) 794 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3947556342"}]': finished 2023-12-16T07:38:47.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: cluster 2023-12-16T07:38:46.372042+0000 mon.a (mon.0) 795 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2023-12-16T07:38:47.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.412295+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.417931+0000 mon.a (mon.0) 797 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.428295+0000 mon.a (mon.0) 798 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:38:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.429281+0000 mgr.y (mgr.24367) 138 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:38:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: cephadm 2023-12-16T07:38:46.430976+0000 mgr.y (mgr.24367) 139 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2023-12-16T07:38:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.431293+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:38:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.431732+0000 mgr.y (mgr.24367) 140 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:38:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.438490+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.443436+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:38:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.444182+0000 mgr.y (mgr.24367) 141 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:38:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.458187+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.460560+0000 mon.a (mon.0) 803 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.461556+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.469866+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.738871+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.7:0/278578181' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3815256908"}]: dispatch 2023-12-16T07:38:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[30436]: audit 2023-12-16T07:38:46.739643+0000 mon.a (mon.0) 806 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3815256908"}]: dispatch 2023-12-16T07:38:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: cluster 2023-12-16T07:38:46.248826+0000 mgr.y (mgr.24367) 137 : cluster [DBG] pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 112 B/s wr, 3 op/s 2023-12-16T07:38:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.371900+0000 mon.a (mon.0) 794 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3947556342"}]': finished 2023-12-16T07:38:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: cluster 2023-12-16T07:38:46.372042+0000 mon.a (mon.0) 795 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2023-12-16T07:38:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.412295+0000 mon.a (mon.0) 796 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.417931+0000 mon.a (mon.0) 797 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.428295+0000 mon.a (mon.0) 798 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:38:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.429281+0000 mgr.y (mgr.24367) 138 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:38:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: cephadm 2023-12-16T07:38:46.430976+0000 mgr.y (mgr.24367) 139 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2023-12-16T07:38:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.431293+0000 mon.a (mon.0) 799 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:38:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.431732+0000 mgr.y (mgr.24367) 140 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:38:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.438490+0000 mon.a (mon.0) 800 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.443436+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:38:47.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.444182+0000 mgr.y (mgr.24367) 141 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:38:47.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.458187+0000 mon.a (mon.0) 802 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.460560+0000 mon.a (mon.0) 803 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:38:47.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.461556+0000 mon.a (mon.0) 804 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:38:47.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.469866+0000 mon.a (mon.0) 805 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:38:47.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.738871+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.7:0/278578181' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3815256908"}]: dispatch 2023-12-16T07:38:47.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:47 smithi007 bash[25009]: audit 2023-12-16T07:38:46.739643+0000 mon.a (mon.0) 806 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3815256908"}]: dispatch 2023-12-16T07:38:48.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:48 smithi007 bash[30436]: audit 2023-12-16T07:38:47.475614+0000 mon.a (mon.0) 807 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3815256908"}]': finished 2023-12-16T07:38:49.202 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:48 smithi007 bash[30436]: cluster 2023-12-16T07:38:47.475893+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2023-12-16T07:38:49.203 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:48 smithi007 bash[30436]: audit 2023-12-16T07:38:47.847634+0000 mon.a (mon.0) 809 : audit [INF] from='client.? 172.21.15.7:0/3274381036' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2722441948"}]: dispatch 2023-12-16T07:38:49.203 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:48 smithi007 bash[25009]: audit 2023-12-16T07:38:47.475614+0000 mon.a (mon.0) 807 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3815256908"}]': finished 2023-12-16T07:38:49.203 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:48 smithi007 bash[25009]: cluster 2023-12-16T07:38:47.475893+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2023-12-16T07:38:49.203 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:48 smithi007 bash[25009]: audit 2023-12-16T07:38:47.847634+0000 mon.a (mon.0) 809 : audit [INF] from='client.? 172.21.15.7:0/3274381036' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2722441948"}]: dispatch 2023-12-16T07:38:49.203 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:48 smithi169 bash[25083]: audit 2023-12-16T07:38:47.475614+0000 mon.a (mon.0) 807 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3815256908"}]': finished 2023-12-16T07:38:49.204 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:48 smithi169 bash[25083]: cluster 2023-12-16T07:38:47.475893+0000 mon.a (mon.0) 808 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2023-12-16T07:38:49.204 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:48 smithi169 bash[25083]: audit 2023-12-16T07:38:47.847634+0000 mon.a (mon.0) 809 : audit [INF] from='client.? 172.21.15.7:0/3274381036' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2722441948"}]: dispatch 2023-12-16T07:38:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:49 smithi007 bash[30436]: cluster 2023-12-16T07:38:48.249577+0000 mgr.y (mgr.24367) 142 : cluster [DBG] pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 170 B/s wr, 1 op/s 2023-12-16T07:38:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:49 smithi007 bash[30436]: audit 2023-12-16T07:38:48.488160+0000 mon.a (mon.0) 810 : audit [INF] from='client.? 172.21.15.7:0/3274381036' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2722441948"}]': finished 2023-12-16T07:38:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:49 smithi007 bash[30436]: cluster 2023-12-16T07:38:48.488263+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2023-12-16T07:38:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:49 smithi007 bash[30436]: audit 2023-12-16T07:38:48.847818+0000 mon.c (mon.1) 27 : audit [INF] from='client.? 172.21.15.7:0/4276772092' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2431018109"}]: dispatch 2023-12-16T07:38:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:49 smithi007 bash[30436]: audit 2023-12-16T07:38:48.848616+0000 mon.a (mon.0) 812 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2431018109"}]: dispatch 2023-12-16T07:38:49.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:49 smithi007 bash[25009]: cluster 2023-12-16T07:38:48.249577+0000 mgr.y (mgr.24367) 142 : cluster [DBG] pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 170 B/s wr, 1 op/s 2023-12-16T07:38:49.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:49 smithi007 bash[25009]: audit 2023-12-16T07:38:48.488160+0000 mon.a (mon.0) 810 : audit [INF] from='client.? 172.21.15.7:0/3274381036' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2722441948"}]': finished 2023-12-16T07:38:49.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:49 smithi007 bash[25009]: cluster 2023-12-16T07:38:48.488263+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2023-12-16T07:38:49.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:49 smithi007 bash[25009]: audit 2023-12-16T07:38:48.847818+0000 mon.c (mon.1) 27 : audit [INF] from='client.? 172.21.15.7:0/4276772092' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2431018109"}]: dispatch 2023-12-16T07:38:49.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:49 smithi007 bash[25009]: audit 2023-12-16T07:38:48.848616+0000 mon.a (mon.0) 812 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2431018109"}]: dispatch 2023-12-16T07:38:49.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:49 smithi169 bash[25083]: cluster 2023-12-16T07:38:48.249577+0000 mgr.y (mgr.24367) 142 : cluster [DBG] pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 170 B/s wr, 1 op/s 2023-12-16T07:38:49.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:49 smithi169 bash[25083]: audit 2023-12-16T07:38:48.488160+0000 mon.a (mon.0) 810 : audit [INF] from='client.? 172.21.15.7:0/3274381036' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2722441948"}]': finished 2023-12-16T07:38:49.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:49 smithi169 bash[25083]: cluster 2023-12-16T07:38:48.488263+0000 mon.a (mon.0) 811 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2023-12-16T07:38:49.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:49 smithi169 bash[25083]: audit 2023-12-16T07:38:48.847818+0000 mon.c (mon.1) 27 : audit [INF] from='client.? 172.21.15.7:0/4276772092' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2431018109"}]: dispatch 2023-12-16T07:38:49.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:49 smithi169 bash[25083]: audit 2023-12-16T07:38:48.848616+0000 mon.a (mon.0) 812 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2431018109"}]: dispatch 2023-12-16T07:38:50.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:50 smithi007 bash[30436]: audit 2023-12-16T07:38:49.495631+0000 mon.a (mon.0) 813 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2431018109"}]': finished 2023-12-16T07:38:50.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:50 smithi007 bash[30436]: cluster 2023-12-16T07:38:49.495747+0000 mon.a (mon.0) 814 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2023-12-16T07:38:50.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:50 smithi007 bash[30436]: audit 2023-12-16T07:38:49.872042+0000 mon.a (mon.0) 815 : audit [INF] from='client.? 172.21.15.7:0/3776734446' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/847724836"}]: dispatch 2023-12-16T07:38:50.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:50 smithi007 bash[30436]: audit 2023-12-16T07:38:50.209168+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2023-12-16T07:38:50.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:50 smithi007 bash[30436]: audit 2023-12-16T07:38:50.209815+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2023-12-16T07:38:50.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:50 smithi007 bash[30436]: audit 2023-12-16T07:38:50.210293+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2023-12-16T07:38:50.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:50 smithi007 bash[30436]: audit 2023-12-16T07:38:50.210784+0000 mon.a (mon.0) 819 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2023-12-16T07:38:50.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:50 smithi007 bash[25009]: audit 2023-12-16T07:38:49.495631+0000 mon.a (mon.0) 813 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2431018109"}]': finished 2023-12-16T07:38:50.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:50 smithi007 bash[25009]: cluster 2023-12-16T07:38:49.495747+0000 mon.a (mon.0) 814 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2023-12-16T07:38:50.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:50 smithi007 bash[25009]: audit 2023-12-16T07:38:49.872042+0000 mon.a (mon.0) 815 : audit [INF] from='client.? 172.21.15.7:0/3776734446' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/847724836"}]: dispatch 2023-12-16T07:38:50.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:50 smithi007 bash[25009]: audit 2023-12-16T07:38:50.209168+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2023-12-16T07:38:50.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:50 smithi007 bash[25009]: audit 2023-12-16T07:38:50.209815+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2023-12-16T07:38:50.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:50 smithi007 bash[25009]: audit 2023-12-16T07:38:50.210293+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2023-12-16T07:38:50.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:50 smithi007 bash[25009]: audit 2023-12-16T07:38:50.210784+0000 mon.a (mon.0) 819 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2023-12-16T07:38:50.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:50 smithi169 bash[25083]: audit 2023-12-16T07:38:49.495631+0000 mon.a (mon.0) 813 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2431018109"}]': finished 2023-12-16T07:38:50.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:50 smithi169 bash[25083]: cluster 2023-12-16T07:38:49.495747+0000 mon.a (mon.0) 814 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2023-12-16T07:38:50.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:50 smithi169 bash[25083]: audit 2023-12-16T07:38:49.872042+0000 mon.a (mon.0) 815 : audit [INF] from='client.? 172.21.15.7:0/3776734446' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/847724836"}]: dispatch 2023-12-16T07:38:50.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:50 smithi169 bash[25083]: audit 2023-12-16T07:38:50.209168+0000 mon.a (mon.0) 816 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]: dispatch 2023-12-16T07:38:50.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:50 smithi169 bash[25083]: audit 2023-12-16T07:38:50.209815+0000 mon.a (mon.0) 817 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2023-12-16T07:38:50.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:50 smithi169 bash[25083]: audit 2023-12-16T07:38:50.210293+0000 mon.a (mon.0) 818 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2023-12-16T07:38:50.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:50 smithi169 bash[25083]: audit 2023-12-16T07:38:50.210784+0000 mon.a (mon.0) 819 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2023-12-16T07:38:51.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[25009]: cluster 2023-12-16T07:38:50.250313+0000 mgr.y (mgr.24367) 143 : cluster [DBG] pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:38:51.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[25009]: audit 2023-12-16T07:38:50.508868+0000 mon.a (mon.0) 820 : audit [INF] from='client.? 172.21.15.7:0/3776734446' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/847724836"}]': finished 2023-12-16T07:38:51.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[25009]: audit 2023-12-16T07:38:50.508974+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2023-12-16T07:38:51.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[25009]: audit 2023-12-16T07:38:50.509123+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2023-12-16T07:38:51.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[25009]: audit 2023-12-16T07:38:50.509228+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2023-12-16T07:38:51.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[25009]: audit 2023-12-16T07:38:50.509349+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2023-12-16T07:38:51.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[25009]: cluster 2023-12-16T07:38:50.509446+0000 mon.a (mon.0) 825 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2023-12-16T07:38:51.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[25009]: audit 2023-12-16T07:38:50.529622+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:38:51.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[25009]: audit 2023-12-16T07:38:50.541595+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:38:51.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[25009]: audit 2023-12-16T07:38:50.879825+0000 mon.c (mon.1) 28 : audit [INF] from='client.? 172.21.15.7:0/3327308804' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2086785434"}]: dispatch 2023-12-16T07:38:51.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[25009]: audit 2023-12-16T07:38:50.880898+0000 mon.a (mon.0) 828 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2086785434"}]: dispatch 2023-12-16T07:38:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[30436]: cluster 2023-12-16T07:38:50.250313+0000 mgr.y (mgr.24367) 143 : cluster [DBG] pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:38:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[30436]: audit 2023-12-16T07:38:50.508868+0000 mon.a (mon.0) 820 : audit [INF] from='client.? 172.21.15.7:0/3776734446' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/847724836"}]': finished 2023-12-16T07:38:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[30436]: audit 2023-12-16T07:38:50.508974+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2023-12-16T07:38:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[30436]: audit 2023-12-16T07:38:50.509123+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2023-12-16T07:38:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[30436]: audit 2023-12-16T07:38:50.509228+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2023-12-16T07:38:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[30436]: audit 2023-12-16T07:38:50.509349+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2023-12-16T07:38:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[30436]: cluster 2023-12-16T07:38:50.509446+0000 mon.a (mon.0) 825 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2023-12-16T07:38:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[30436]: audit 2023-12-16T07:38:50.529622+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:38:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[30436]: audit 2023-12-16T07:38:50.541595+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:38:51.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[30436]: audit 2023-12-16T07:38:50.879825+0000 mon.c (mon.1) 28 : audit [INF] from='client.? 172.21.15.7:0/3327308804' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2086785434"}]: dispatch 2023-12-16T07:38:51.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:51 smithi007 bash[30436]: audit 2023-12-16T07:38:50.880898+0000 mon.a (mon.0) 828 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2086785434"}]: dispatch 2023-12-16T07:38:51.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:51 smithi169 bash[25083]: cluster 2023-12-16T07:38:50.250313+0000 mgr.y (mgr.24367) 143 : cluster [DBG] pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:38:51.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:51 smithi169 bash[25083]: audit 2023-12-16T07:38:50.508868+0000 mon.a (mon.0) 820 : audit [INF] from='client.? 172.21.15.7:0/3776734446' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/847724836"}]': finished 2023-12-16T07:38:51.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:51 smithi169 bash[25083]: audit 2023-12-16T07:38:50.508974+0000 mon.a (mon.0) 821 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [7, 2]}]': finished 2023-12-16T07:38:51.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:51 smithi169 bash[25083]: audit 2023-12-16T07:38:50.509123+0000 mon.a (mon.0) 822 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2023-12-16T07:38:51.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:51 smithi169 bash[25083]: audit 2023-12-16T07:38:50.509228+0000 mon.a (mon.0) 823 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2023-12-16T07:38:51.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:51 smithi169 bash[25083]: audit 2023-12-16T07:38:50.509349+0000 mon.a (mon.0) 824 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2023-12-16T07:38:51.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:51 smithi169 bash[25083]: cluster 2023-12-16T07:38:50.509446+0000 mon.a (mon.0) 825 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2023-12-16T07:38:51.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:51 smithi169 bash[25083]: audit 2023-12-16T07:38:50.529622+0000 mon.a (mon.0) 826 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:38:51.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:51 smithi169 bash[25083]: audit 2023-12-16T07:38:50.541595+0000 mon.a (mon.0) 827 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:38:51.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:51 smithi169 bash[25083]: audit 2023-12-16T07:38:50.879825+0000 mon.c (mon.1) 28 : audit [INF] from='client.? 172.21.15.7:0/3327308804' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2086785434"}]: dispatch 2023-12-16T07:38:51.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:51 smithi169 bash[25083]: audit 2023-12-16T07:38:50.880898+0000 mon.a (mon.0) 828 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2086785434"}]: dispatch 2023-12-16T07:38:52.521 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:38:52 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:38:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:38:52.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:52 smithi007 bash[30436]: audit 2023-12-16T07:38:51.509893+0000 mon.a (mon.0) 829 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2086785434"}]': finished 2023-12-16T07:38:52.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:52 smithi007 bash[30436]: cluster 2023-12-16T07:38:51.510100+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2023-12-16T07:38:52.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:52 smithi007 bash[30436]: audit 2023-12-16T07:38:51.876139+0000 mon.c (mon.1) 29 : audit [INF] from='client.? 172.21.15.7:0/3039317474' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2454016267"}]: dispatch 2023-12-16T07:38:52.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:52 smithi007 bash[30436]: audit 2023-12-16T07:38:51.877127+0000 mon.a (mon.0) 831 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2454016267"}]: dispatch 2023-12-16T07:38:52.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:52 smithi007 bash[25009]: audit 2023-12-16T07:38:51.509893+0000 mon.a (mon.0) 829 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2086785434"}]': finished 2023-12-16T07:38:52.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:52 smithi007 bash[25009]: cluster 2023-12-16T07:38:51.510100+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2023-12-16T07:38:52.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:52 smithi007 bash[25009]: audit 2023-12-16T07:38:51.876139+0000 mon.c (mon.1) 29 : audit [INF] from='client.? 172.21.15.7:0/3039317474' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2454016267"}]: dispatch 2023-12-16T07:38:52.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:52 smithi007 bash[25009]: audit 2023-12-16T07:38:51.877127+0000 mon.a (mon.0) 831 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2454016267"}]: dispatch 2023-12-16T07:38:52.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:52 smithi169 bash[25083]: audit 2023-12-16T07:38:51.509893+0000 mon.a (mon.0) 829 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2086785434"}]': finished 2023-12-16T07:38:52.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:52 smithi169 bash[25083]: cluster 2023-12-16T07:38:51.510100+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2023-12-16T07:38:52.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:52 smithi169 bash[25083]: audit 2023-12-16T07:38:51.876139+0000 mon.c (mon.1) 29 : audit [INF] from='client.? 172.21.15.7:0/3039317474' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2454016267"}]: dispatch 2023-12-16T07:38:52.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:52 smithi169 bash[25083]: audit 2023-12-16T07:38:51.877127+0000 mon.a (mon.0) 831 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2454016267"}]: dispatch 2023-12-16T07:38:53.816 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:38:53 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:38:53] "GET /metrics HTTP/1.1" 200 214917 "" "Prometheus/2.33.4" 2023-12-16T07:38:53.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:38:53 smithi007 bash[49264]: level=error ts=2023-12-16T07:38:53.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:38:53.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:53 smithi007 bash[25009]: cluster 2023-12-16T07:38:52.251140+0000 mgr.y (mgr.24367) 144 : cluster [DBG] pgmap v119: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:38:53.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:53 smithi007 bash[25009]: audit 2023-12-16T07:38:52.532240+0000 mon.a (mon.0) 832 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2454016267"}]': finished 2023-12-16T07:38:53.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:53 smithi007 bash[25009]: cluster 2023-12-16T07:38:52.532471+0000 mon.a (mon.0) 833 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2023-12-16T07:38:53.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:53 smithi007 bash[25009]: audit 2023-12-16T07:38:52.900725+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.7:0/2831506890' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/847724836"}]: dispatch 2023-12-16T07:38:53.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:53 smithi007 bash[25009]: audit 2023-12-16T07:38:52.901739+0000 mon.a (mon.0) 834 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/847724836"}]: dispatch 2023-12-16T07:38:53.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:53 smithi007 bash[30436]: cluster 2023-12-16T07:38:52.251140+0000 mgr.y (mgr.24367) 144 : cluster [DBG] pgmap v119: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:38:53.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:53 smithi007 bash[30436]: audit 2023-12-16T07:38:52.532240+0000 mon.a (mon.0) 832 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2454016267"}]': finished 2023-12-16T07:38:53.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:53 smithi007 bash[30436]: cluster 2023-12-16T07:38:52.532471+0000 mon.a (mon.0) 833 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2023-12-16T07:38:53.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:53 smithi007 bash[30436]: audit 2023-12-16T07:38:52.900725+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.7:0/2831506890' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/847724836"}]: dispatch 2023-12-16T07:38:53.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:53 smithi007 bash[30436]: audit 2023-12-16T07:38:52.901739+0000 mon.a (mon.0) 834 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/847724836"}]: dispatch 2023-12-16T07:38:53.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:53 smithi169 bash[25083]: cluster 2023-12-16T07:38:52.251140+0000 mgr.y (mgr.24367) 144 : cluster [DBG] pgmap v119: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:38:53.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:53 smithi169 bash[25083]: audit 2023-12-16T07:38:52.532240+0000 mon.a (mon.0) 832 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2454016267"}]': finished 2023-12-16T07:38:53.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:53 smithi169 bash[25083]: cluster 2023-12-16T07:38:52.532471+0000 mon.a (mon.0) 833 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2023-12-16T07:38:53.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:53 smithi169 bash[25083]: audit 2023-12-16T07:38:52.900725+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.7:0/2831506890' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/847724836"}]: dispatch 2023-12-16T07:38:53.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:53 smithi169 bash[25083]: audit 2023-12-16T07:38:52.901739+0000 mon.a (mon.0) 834 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/847724836"}]: dispatch 2023-12-16T07:38:54.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:54 smithi007 bash[30436]: audit 2023-12-16T07:38:53.543144+0000 mon.a (mon.0) 835 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/847724836"}]': finished 2023-12-16T07:38:54.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:54 smithi007 bash[30436]: cluster 2023-12-16T07:38:53.543256+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2023-12-16T07:38:54.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:54 smithi007 bash[30436]: audit 2023-12-16T07:38:53.596323+0000 mgr.y (mgr.24367) 145 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:38:54.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:54 smithi007 bash[30436]: audit 2023-12-16T07:38:53.909338+0000 mon.c (mon.1) 31 : audit [INF] from='client.? 172.21.15.7:0/1097895379' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2442196965"}]: dispatch 2023-12-16T07:38:54.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:54 smithi007 bash[30436]: audit 2023-12-16T07:38:53.910481+0000 mon.a (mon.0) 837 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2442196965"}]: dispatch 2023-12-16T07:38:54.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:54 smithi007 bash[25009]: audit 2023-12-16T07:38:53.543144+0000 mon.a (mon.0) 835 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/847724836"}]': finished 2023-12-16T07:38:54.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:54 smithi007 bash[25009]: cluster 2023-12-16T07:38:53.543256+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2023-12-16T07:38:54.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:54 smithi007 bash[25009]: audit 2023-12-16T07:38:53.596323+0000 mgr.y (mgr.24367) 145 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:38:54.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:54 smithi007 bash[25009]: audit 2023-12-16T07:38:53.909338+0000 mon.c (mon.1) 31 : audit [INF] from='client.? 172.21.15.7:0/1097895379' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2442196965"}]: dispatch 2023-12-16T07:38:54.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:54 smithi007 bash[25009]: audit 2023-12-16T07:38:53.910481+0000 mon.a (mon.0) 837 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2442196965"}]: dispatch 2023-12-16T07:38:54.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:54 smithi169 bash[25083]: audit 2023-12-16T07:38:53.543144+0000 mon.a (mon.0) 835 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/847724836"}]': finished 2023-12-16T07:38:54.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:54 smithi169 bash[25083]: cluster 2023-12-16T07:38:53.543256+0000 mon.a (mon.0) 836 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2023-12-16T07:38:54.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:54 smithi169 bash[25083]: audit 2023-12-16T07:38:53.596323+0000 mgr.y (mgr.24367) 145 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:38:54.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:54 smithi169 bash[25083]: audit 2023-12-16T07:38:53.909338+0000 mon.c (mon.1) 31 : audit [INF] from='client.? 172.21.15.7:0/1097895379' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2442196965"}]: dispatch 2023-12-16T07:38:54.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:54 smithi169 bash[25083]: audit 2023-12-16T07:38:53.910481+0000 mon.a (mon.0) 837 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2442196965"}]: dispatch 2023-12-16T07:38:55.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:55 smithi007 bash[25009]: cluster 2023-12-16T07:38:54.251895+0000 mgr.y (mgr.24367) 146 : cluster [DBG] pgmap v122: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:38:55.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:55 smithi007 bash[25009]: audit 2023-12-16T07:38:54.552497+0000 mon.a (mon.0) 838 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2442196965"}]': finished 2023-12-16T07:38:55.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:55 smithi007 bash[25009]: cluster 2023-12-16T07:38:54.552639+0000 mon.a (mon.0) 839 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2023-12-16T07:38:55.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:55 smithi007 bash[25009]: audit 2023-12-16T07:38:54.922766+0000 mon.a (mon.0) 840 : audit [INF] from='client.? 172.21.15.7:0/1823206020' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1208557731"}]: dispatch 2023-12-16T07:38:55.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:55 smithi007 bash[30436]: cluster 2023-12-16T07:38:54.251895+0000 mgr.y (mgr.24367) 146 : cluster [DBG] pgmap v122: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:38:55.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:55 smithi007 bash[30436]: audit 2023-12-16T07:38:54.552497+0000 mon.a (mon.0) 838 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2442196965"}]': finished 2023-12-16T07:38:55.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:55 smithi007 bash[30436]: cluster 2023-12-16T07:38:54.552639+0000 mon.a (mon.0) 839 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2023-12-16T07:38:55.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:55 smithi007 bash[30436]: audit 2023-12-16T07:38:54.922766+0000 mon.a (mon.0) 840 : audit [INF] from='client.? 172.21.15.7:0/1823206020' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1208557731"}]: dispatch 2023-12-16T07:38:55.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:55 smithi169 bash[25083]: cluster 2023-12-16T07:38:54.251895+0000 mgr.y (mgr.24367) 146 : cluster [DBG] pgmap v122: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:38:55.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:55 smithi169 bash[25083]: audit 2023-12-16T07:38:54.552497+0000 mon.a (mon.0) 838 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2442196965"}]': finished 2023-12-16T07:38:55.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:55 smithi169 bash[25083]: cluster 2023-12-16T07:38:54.552639+0000 mon.a (mon.0) 839 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2023-12-16T07:38:55.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:55 smithi169 bash[25083]: audit 2023-12-16T07:38:54.922766+0000 mon.a (mon.0) 840 : audit [INF] from='client.? 172.21.15.7:0/1823206020' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1208557731"}]: dispatch 2023-12-16T07:38:56.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:56 smithi007 bash[25009]: audit 2023-12-16T07:38:55.559768+0000 mon.a (mon.0) 841 : audit [INF] from='client.? 172.21.15.7:0/1823206020' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1208557731"}]': finished 2023-12-16T07:38:56.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:56 smithi007 bash[25009]: cluster 2023-12-16T07:38:55.559871+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2023-12-16T07:38:56.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:56 smithi007 bash[25009]: audit 2023-12-16T07:38:55.937777+0000 mon.c (mon.1) 32 : audit [INF] from='client.? 172.21.15.7:0/2849786099' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2454016267"}]: dispatch 2023-12-16T07:38:56.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:56 smithi007 bash[25009]: audit 2023-12-16T07:38:55.938891+0000 mon.a (mon.0) 843 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2454016267"}]: dispatch 2023-12-16T07:38:56.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:56 smithi007 bash[30436]: audit 2023-12-16T07:38:55.559768+0000 mon.a (mon.0) 841 : audit [INF] from='client.? 172.21.15.7:0/1823206020' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1208557731"}]': finished 2023-12-16T07:38:56.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:56 smithi007 bash[30436]: cluster 2023-12-16T07:38:55.559871+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2023-12-16T07:38:56.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:56 smithi007 bash[30436]: audit 2023-12-16T07:38:55.937777+0000 mon.c (mon.1) 32 : audit [INF] from='client.? 172.21.15.7:0/2849786099' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2454016267"}]: dispatch 2023-12-16T07:38:56.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:56 smithi007 bash[30436]: audit 2023-12-16T07:38:55.938891+0000 mon.a (mon.0) 843 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2454016267"}]: dispatch 2023-12-16T07:38:56.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:56 smithi169 bash[25083]: audit 2023-12-16T07:38:55.559768+0000 mon.a (mon.0) 841 : audit [INF] from='client.? 172.21.15.7:0/1823206020' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1208557731"}]': finished 2023-12-16T07:38:56.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:56 smithi169 bash[25083]: cluster 2023-12-16T07:38:55.559871+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2023-12-16T07:38:56.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:56 smithi169 bash[25083]: audit 2023-12-16T07:38:55.937777+0000 mon.c (mon.1) 32 : audit [INF] from='client.? 172.21.15.7:0/2849786099' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2454016267"}]: dispatch 2023-12-16T07:38:56.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:56 smithi169 bash[25083]: audit 2023-12-16T07:38:55.938891+0000 mon.a (mon.0) 843 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2454016267"}]: dispatch 2023-12-16T07:38:57.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:57 smithi169 bash[25083]: cluster 2023-12-16T07:38:56.254703+0000 mgr.y (mgr.24367) 147 : cluster [DBG] pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11 B/s, 0 objects/s recovering 2023-12-16T07:38:57.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:57 smithi169 bash[25083]: audit 2023-12-16T07:38:56.573737+0000 mon.a (mon.0) 844 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2454016267"}]': finished 2023-12-16T07:38:57.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:57 smithi169 bash[25083]: cluster 2023-12-16T07:38:56.573882+0000 mon.a (mon.0) 845 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2023-12-16T07:38:57.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:57 smithi169 bash[25083]: audit 2023-12-16T07:38:56.946374+0000 mon.a (mon.0) 846 : audit [INF] from='client.? 172.21.15.7:0/2844911165' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2162341476"}]: dispatch 2023-12-16T07:38:58.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:57 smithi007 bash[30436]: cluster 2023-12-16T07:38:56.254703+0000 mgr.y (mgr.24367) 147 : cluster [DBG] pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11 B/s, 0 objects/s recovering 2023-12-16T07:38:58.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:57 smithi007 bash[30436]: audit 2023-12-16T07:38:56.573737+0000 mon.a (mon.0) 844 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2454016267"}]': finished 2023-12-16T07:38:58.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:57 smithi007 bash[30436]: cluster 2023-12-16T07:38:56.573882+0000 mon.a (mon.0) 845 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2023-12-16T07:38:58.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:57 smithi007 bash[30436]: audit 2023-12-16T07:38:56.946374+0000 mon.a (mon.0) 846 : audit [INF] from='client.? 172.21.15.7:0/2844911165' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2162341476"}]: dispatch 2023-12-16T07:38:58.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:57 smithi007 bash[25009]: cluster 2023-12-16T07:38:56.254703+0000 mgr.y (mgr.24367) 147 : cluster [DBG] pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11 B/s, 0 objects/s recovering 2023-12-16T07:38:58.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:57 smithi007 bash[25009]: audit 2023-12-16T07:38:56.573737+0000 mon.a (mon.0) 844 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2454016267"}]': finished 2023-12-16T07:38:58.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:57 smithi007 bash[25009]: cluster 2023-12-16T07:38:56.573882+0000 mon.a (mon.0) 845 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2023-12-16T07:38:58.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:57 smithi007 bash[25009]: audit 2023-12-16T07:38:56.946374+0000 mon.a (mon.0) 846 : audit [INF] from='client.? 172.21.15.7:0/2844911165' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2162341476"}]: dispatch 2023-12-16T07:38:58.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:58 smithi169 bash[25083]: audit 2023-12-16T07:38:57.583423+0000 mon.a (mon.0) 847 : audit [INF] from='client.? 172.21.15.7:0/2844911165' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2162341476"}]': finished 2023-12-16T07:38:58.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:58 smithi169 bash[25083]: cluster 2023-12-16T07:38:57.583558+0000 mon.a (mon.0) 848 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2023-12-16T07:38:58.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:58 smithi169 bash[25083]: audit 2023-12-16T07:38:57.948904+0000 mon.a (mon.0) 849 : audit [INF] from='client.? 172.21.15.7:0/3901408268' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3673321318"}]: dispatch 2023-12-16T07:38:59.066 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:58 smithi007 bash[25009]: audit 2023-12-16T07:38:57.583423+0000 mon.a (mon.0) 847 : audit [INF] from='client.? 172.21.15.7:0/2844911165' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2162341476"}]': finished 2023-12-16T07:38:59.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:58 smithi007 bash[25009]: cluster 2023-12-16T07:38:57.583558+0000 mon.a (mon.0) 848 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2023-12-16T07:38:59.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:58 smithi007 bash[25009]: audit 2023-12-16T07:38:57.948904+0000 mon.a (mon.0) 849 : audit [INF] from='client.? 172.21.15.7:0/3901408268' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3673321318"}]: dispatch 2023-12-16T07:38:59.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:58 smithi007 bash[30436]: audit 2023-12-16T07:38:57.583423+0000 mon.a (mon.0) 847 : audit [INF] from='client.? 172.21.15.7:0/2844911165' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2162341476"}]': finished 2023-12-16T07:38:59.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:58 smithi007 bash[30436]: cluster 2023-12-16T07:38:57.583558+0000 mon.a (mon.0) 848 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2023-12-16T07:38:59.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:58 smithi007 bash[30436]: audit 2023-12-16T07:38:57.948904+0000 mon.a (mon.0) 849 : audit [INF] from='client.? 172.21.15.7:0/3901408268' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3673321318"}]: dispatch 2023-12-16T07:38:59.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:59 smithi169 bash[25083]: cluster 2023-12-16T07:38:58.255533+0000 mgr.y (mgr.24367) 148 : cluster [DBG] pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11 B/s, 0 objects/s recovering 2023-12-16T07:38:59.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:59 smithi169 bash[25083]: audit 2023-12-16T07:38:58.591959+0000 mon.a (mon.0) 850 : audit [INF] from='client.? 172.21.15.7:0/3901408268' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3673321318"}]': finished 2023-12-16T07:38:59.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:59 smithi169 bash[25083]: cluster 2023-12-16T07:38:58.592089+0000 mon.a (mon.0) 851 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2023-12-16T07:38:59.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:38:59 smithi169 bash[25083]: audit 2023-12-16T07:38:58.971336+0000 mon.a (mon.0) 852 : audit [INF] from='client.? 172.21.15.7:0/2967437442' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3972277094"}]: dispatch 2023-12-16T07:39:00.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:59 smithi007 bash[30436]: cluster 2023-12-16T07:38:58.255533+0000 mgr.y (mgr.24367) 148 : cluster [DBG] pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11 B/s, 0 objects/s recovering 2023-12-16T07:39:00.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:59 smithi007 bash[30436]: audit 2023-12-16T07:38:58.591959+0000 mon.a (mon.0) 850 : audit [INF] from='client.? 172.21.15.7:0/3901408268' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3673321318"}]': finished 2023-12-16T07:39:00.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:59 smithi007 bash[30436]: cluster 2023-12-16T07:38:58.592089+0000 mon.a (mon.0) 851 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2023-12-16T07:39:00.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:38:59 smithi007 bash[30436]: audit 2023-12-16T07:38:58.971336+0000 mon.a (mon.0) 852 : audit [INF] from='client.? 172.21.15.7:0/2967437442' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3972277094"}]: dispatch 2023-12-16T07:39:00.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:59 smithi007 bash[25009]: cluster 2023-12-16T07:38:58.255533+0000 mgr.y (mgr.24367) 148 : cluster [DBG] pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11 B/s, 0 objects/s recovering 2023-12-16T07:39:00.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:59 smithi007 bash[25009]: audit 2023-12-16T07:38:58.591959+0000 mon.a (mon.0) 850 : audit [INF] from='client.? 172.21.15.7:0/3901408268' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3673321318"}]': finished 2023-12-16T07:39:00.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:59 smithi007 bash[25009]: cluster 2023-12-16T07:38:58.592089+0000 mon.a (mon.0) 851 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2023-12-16T07:39:00.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:38:59 smithi007 bash[25009]: audit 2023-12-16T07:38:58.971336+0000 mon.a (mon.0) 852 : audit [INF] from='client.? 172.21.15.7:0/2967437442' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3972277094"}]: dispatch 2023-12-16T07:39:00.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:00 smithi169 bash[25083]: audit 2023-12-16T07:38:59.603099+0000 mon.a (mon.0) 853 : audit [INF] from='client.? 172.21.15.7:0/2967437442' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3972277094"}]': finished 2023-12-16T07:39:00.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:00 smithi169 bash[25083]: cluster 2023-12-16T07:38:59.603273+0000 mon.a (mon.0) 854 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2023-12-16T07:39:00.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:00 smithi169 bash[25083]: audit 2023-12-16T07:38:59.976192+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.7:0/339121917' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3086453482"}]: dispatch 2023-12-16T07:39:00.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:00 smithi169 bash[25083]: audit 2023-12-16T07:38:59.977235+0000 mon.a (mon.0) 855 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3086453482"}]: dispatch 2023-12-16T07:39:01.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:00 smithi007 bash[25009]: audit 2023-12-16T07:38:59.603099+0000 mon.a (mon.0) 853 : audit [INF] from='client.? 172.21.15.7:0/2967437442' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3972277094"}]': finished 2023-12-16T07:39:01.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:00 smithi007 bash[25009]: cluster 2023-12-16T07:38:59.603273+0000 mon.a (mon.0) 854 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2023-12-16T07:39:01.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:00 smithi007 bash[25009]: audit 2023-12-16T07:38:59.976192+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.7:0/339121917' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3086453482"}]: dispatch 2023-12-16T07:39:01.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:00 smithi007 bash[25009]: audit 2023-12-16T07:38:59.977235+0000 mon.a (mon.0) 855 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3086453482"}]: dispatch 2023-12-16T07:39:01.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:00 smithi007 bash[30436]: audit 2023-12-16T07:38:59.603099+0000 mon.a (mon.0) 853 : audit [INF] from='client.? 172.21.15.7:0/2967437442' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3972277094"}]': finished 2023-12-16T07:39:01.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:00 smithi007 bash[30436]: cluster 2023-12-16T07:38:59.603273+0000 mon.a (mon.0) 854 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2023-12-16T07:39:01.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:00 smithi007 bash[30436]: audit 2023-12-16T07:38:59.976192+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.7:0/339121917' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3086453482"}]: dispatch 2023-12-16T07:39:01.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:00 smithi007 bash[30436]: audit 2023-12-16T07:38:59.977235+0000 mon.a (mon.0) 855 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3086453482"}]: dispatch 2023-12-16T07:39:01.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:01 smithi169 bash[25083]: cluster 2023-12-16T07:39:00.256375+0000 mgr.y (mgr.24367) 149 : cluster [DBG] pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:39:01.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:01 smithi169 bash[25083]: audit 2023-12-16T07:39:00.613707+0000 mon.a (mon.0) 856 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3086453482"}]': finished 2023-12-16T07:39:01.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:01 smithi169 bash[25083]: cluster 2023-12-16T07:39:00.613847+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2023-12-16T07:39:02.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:01 smithi007 bash[25009]: cluster 2023-12-16T07:39:00.256375+0000 mgr.y (mgr.24367) 149 : cluster [DBG] pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:39:02.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:01 smithi007 bash[25009]: audit 2023-12-16T07:39:00.613707+0000 mon.a (mon.0) 856 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3086453482"}]': finished 2023-12-16T07:39:02.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:01 smithi007 bash[25009]: cluster 2023-12-16T07:39:00.613847+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2023-12-16T07:39:02.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:01 smithi007 bash[30436]: cluster 2023-12-16T07:39:00.256375+0000 mgr.y (mgr.24367) 149 : cluster [DBG] pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:39:02.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:01 smithi007 bash[30436]: audit 2023-12-16T07:39:00.613707+0000 mon.a (mon.0) 856 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3086453482"}]': finished 2023-12-16T07:39:02.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:01 smithi007 bash[30436]: cluster 2023-12-16T07:39:00.613847+0000 mon.a (mon.0) 857 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2023-12-16T07:39:02.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:39:02 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:39:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:39:03.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:03 smithi169 bash[25083]: cluster 2023-12-16T07:39:02.258157+0000 mgr.y (mgr.24367) 150 : cluster [DBG] pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:39:03.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:03 smithi007 bash[25009]: cluster 2023-12-16T07:39:02.258157+0000 mgr.y (mgr.24367) 150 : cluster [DBG] pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:39:03.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:03 smithi007 bash[30436]: cluster 2023-12-16T07:39:02.258157+0000 mgr.y (mgr.24367) 150 : cluster [DBG] pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:39:04.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:39:03 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:39:03] "GET /metrics HTTP/1.1" 200 214917 "" "Prometheus/2.33.4" 2023-12-16T07:39:04.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:04 smithi169 bash[25083]: audit 2023-12-16T07:39:03.599673+0000 mgr.y (mgr.24367) 151 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:04.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:04 smithi007 bash[25009]: audit 2023-12-16T07:39:03.599673+0000 mgr.y (mgr.24367) 151 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:04.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:04 smithi007 bash[30436]: audit 2023-12-16T07:39:03.599673+0000 mgr.y (mgr.24367) 151 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:05.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:05 smithi169 bash[25083]: cluster 2023-12-16T07:39:04.259032+0000 mgr.y (mgr.24367) 152 : cluster [DBG] pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:05.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:05 smithi007 bash[25009]: cluster 2023-12-16T07:39:04.259032+0000 mgr.y (mgr.24367) 152 : cluster [DBG] pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:05.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:05 smithi007 bash[30436]: cluster 2023-12-16T07:39:04.259032+0000 mgr.y (mgr.24367) 152 : cluster [DBG] pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:07.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:07 smithi169 bash[25083]: cluster 2023-12-16T07:39:06.260876+0000 mgr.y (mgr.24367) 153 : cluster [DBG] pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:39:07.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:07 smithi007 bash[25009]: cluster 2023-12-16T07:39:06.260876+0000 mgr.y (mgr.24367) 153 : cluster [DBG] pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:39:07.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:07 smithi007 bash[30436]: cluster 2023-12-16T07:39:06.260876+0000 mgr.y (mgr.24367) 153 : cluster [DBG] pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:39:09.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:09 smithi169 bash[25083]: cluster 2023-12-16T07:39:08.261664+0000 mgr.y (mgr.24367) 154 : cluster [DBG] pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:09.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:09 smithi007 bash[25009]: cluster 2023-12-16T07:39:08.261664+0000 mgr.y (mgr.24367) 154 : cluster [DBG] pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:09.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:09 smithi007 bash[30436]: cluster 2023-12-16T07:39:08.261664+0000 mgr.y (mgr.24367) 154 : cluster [DBG] pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:11.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:11 smithi169 bash[25083]: cluster 2023-12-16T07:39:10.262516+0000 mgr.y (mgr.24367) 155 : cluster [DBG] pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:39:11.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:11 smithi007 bash[30436]: cluster 2023-12-16T07:39:10.262516+0000 mgr.y (mgr.24367) 155 : cluster [DBG] pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:39:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:11 smithi007 bash[25009]: cluster 2023-12-16T07:39:10.262516+0000 mgr.y (mgr.24367) 155 : cluster [DBG] pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:39:12.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:39:12 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:39:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:39:13.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:13 smithi169 bash[25083]: cluster 2023-12-16T07:39:12.264333+0000 mgr.y (mgr.24367) 156 : cluster [DBG] pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:39:13.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:13 smithi007 bash[30436]: cluster 2023-12-16T07:39:12.264333+0000 mgr.y (mgr.24367) 156 : cluster [DBG] pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:39:13.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:13 smithi007 bash[25009]: cluster 2023-12-16T07:39:12.264333+0000 mgr.y (mgr.24367) 156 : cluster [DBG] pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:39:14.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:39:13 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:39:13] "GET /metrics HTTP/1.1" 200 214957 "" "Prometheus/2.33.4" 2023-12-16T07:39:14.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:14 smithi169 bash[25083]: audit 2023-12-16T07:39:13.610981+0000 mgr.y (mgr.24367) 157 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:14.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:14 smithi007 bash[25009]: audit 2023-12-16T07:39:13.610981+0000 mgr.y (mgr.24367) 157 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:14.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:14 smithi007 bash[30436]: audit 2023-12-16T07:39:13.610981+0000 mgr.y (mgr.24367) 157 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:15.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:15 smithi169 bash[25083]: cluster 2023-12-16T07:39:14.265091+0000 mgr.y (mgr.24367) 158 : cluster [DBG] pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:15.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:15 smithi007 bash[25009]: cluster 2023-12-16T07:39:14.265091+0000 mgr.y (mgr.24367) 158 : cluster [DBG] pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:15.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:15 smithi007 bash[30436]: cluster 2023-12-16T07:39:14.265091+0000 mgr.y (mgr.24367) 158 : cluster [DBG] pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:17.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:17 smithi169 bash[25083]: cluster 2023-12-16T07:39:16.266792+0000 mgr.y (mgr.24367) 159 : cluster [DBG] pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:17.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:17 smithi007 bash[25009]: cluster 2023-12-16T07:39:16.266792+0000 mgr.y (mgr.24367) 159 : cluster [DBG] pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:17.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:17 smithi007 bash[30436]: cluster 2023-12-16T07:39:16.266792+0000 mgr.y (mgr.24367) 159 : cluster [DBG] pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:19.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:19 smithi169 bash[25083]: cluster 2023-12-16T07:39:18.267531+0000 mgr.y (mgr.24367) 160 : cluster [DBG] pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:19.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:19 smithi007 bash[25009]: cluster 2023-12-16T07:39:18.267531+0000 mgr.y (mgr.24367) 160 : cluster [DBG] pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:19.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:19 smithi007 bash[30436]: cluster 2023-12-16T07:39:18.267531+0000 mgr.y (mgr.24367) 160 : cluster [DBG] pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:21 smithi169 bash[25083]: cluster 2023-12-16T07:39:20.268271+0000 mgr.y (mgr.24367) 161 : cluster [DBG] pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:21.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:21 smithi007 bash[25009]: cluster 2023-12-16T07:39:20.268271+0000 mgr.y (mgr.24367) 161 : cluster [DBG] pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:21.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:21 smithi007 bash[30436]: cluster 2023-12-16T07:39:20.268271+0000 mgr.y (mgr.24367) 161 : cluster [DBG] pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:22.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:39:22 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:39:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:39:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:23 smithi169 bash[25083]: cluster 2023-12-16T07:39:22.270109+0000 mgr.y (mgr.24367) 162 : cluster [DBG] pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:23.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:23 smithi007 bash[25009]: cluster 2023-12-16T07:39:22.270109+0000 mgr.y (mgr.24367) 162 : cluster [DBG] pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:23.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:23 smithi007 bash[30436]: cluster 2023-12-16T07:39:22.270109+0000 mgr.y (mgr.24367) 162 : cluster [DBG] pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:24.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:39:23 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:39:23] "GET /metrics HTTP/1.1" 200 214918 "" "Prometheus/2.33.4" 2023-12-16T07:39:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:24 smithi169 bash[25083]: audit 2023-12-16T07:39:23.619153+0000 mgr.y (mgr.24367) 163 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:24.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:24 smithi007 bash[30436]: audit 2023-12-16T07:39:23.619153+0000 mgr.y (mgr.24367) 163 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:24.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:24 smithi007 bash[25009]: audit 2023-12-16T07:39:23.619153+0000 mgr.y (mgr.24367) 163 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:25 smithi169 bash[25083]: cluster 2023-12-16T07:39:24.271075+0000 mgr.y (mgr.24367) 164 : cluster [DBG] pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:25.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:25 smithi007 bash[25009]: cluster 2023-12-16T07:39:24.271075+0000 mgr.y (mgr.24367) 164 : cluster [DBG] pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:25.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:25 smithi007 bash[30436]: cluster 2023-12-16T07:39:24.271075+0000 mgr.y (mgr.24367) 164 : cluster [DBG] pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:27.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:27 smithi169 bash[25083]: cluster 2023-12-16T07:39:26.272796+0000 mgr.y (mgr.24367) 165 : cluster [DBG] pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:27.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:27 smithi007 bash[25009]: cluster 2023-12-16T07:39:26.272796+0000 mgr.y (mgr.24367) 165 : cluster [DBG] pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:27.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:27 smithi007 bash[30436]: cluster 2023-12-16T07:39:26.272796+0000 mgr.y (mgr.24367) 165 : cluster [DBG] pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:29.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:29 smithi169 bash[25083]: cluster 2023-12-16T07:39:28.273585+0000 mgr.y (mgr.24367) 166 : cluster [DBG] pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:29.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:29 smithi007 bash[25009]: cluster 2023-12-16T07:39:28.273585+0000 mgr.y (mgr.24367) 166 : cluster [DBG] pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:29.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:29 smithi007 bash[30436]: cluster 2023-12-16T07:39:28.273585+0000 mgr.y (mgr.24367) 166 : cluster [DBG] pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:31.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:31 smithi169 bash[25083]: cluster 2023-12-16T07:39:30.274317+0000 mgr.y (mgr.24367) 167 : cluster [DBG] pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:31.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:31 smithi007 bash[25009]: cluster 2023-12-16T07:39:30.274317+0000 mgr.y (mgr.24367) 167 : cluster [DBG] pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:31.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:31 smithi007 bash[30436]: cluster 2023-12-16T07:39:30.274317+0000 mgr.y (mgr.24367) 167 : cluster [DBG] pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:32.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:39:32 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:39:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:39:33.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:33 smithi169 bash[25083]: cluster 2023-12-16T07:39:32.276090+0000 mgr.y (mgr.24367) 168 : cluster [DBG] pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:33.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:33 smithi007 bash[25009]: cluster 2023-12-16T07:39:32.276090+0000 mgr.y (mgr.24367) 168 : cluster [DBG] pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:33.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:33 smithi007 bash[30436]: cluster 2023-12-16T07:39:32.276090+0000 mgr.y (mgr.24367) 168 : cluster [DBG] pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:34.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:39:33 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:39:33] "GET /metrics HTTP/1.1" 200 214918 "" "Prometheus/2.33.4" 2023-12-16T07:39:34.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:34 smithi169 bash[25083]: audit 2023-12-16T07:39:33.626418+0000 mgr.y (mgr.24367) 169 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:34.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:34 smithi007 bash[25009]: audit 2023-12-16T07:39:33.626418+0000 mgr.y (mgr.24367) 169 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:34.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:34 smithi007 bash[30436]: audit 2023-12-16T07:39:33.626418+0000 mgr.y (mgr.24367) 169 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:35.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:35 smithi169 bash[25083]: cluster 2023-12-16T07:39:34.276980+0000 mgr.y (mgr.24367) 170 : cluster [DBG] pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:35.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:35 smithi007 bash[25009]: cluster 2023-12-16T07:39:34.276980+0000 mgr.y (mgr.24367) 170 : cluster [DBG] pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:35.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:35 smithi007 bash[30436]: cluster 2023-12-16T07:39:34.276980+0000 mgr.y (mgr.24367) 170 : cluster [DBG] pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:37.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:37 smithi169 bash[25083]: cluster 2023-12-16T07:39:36.278756+0000 mgr.y (mgr.24367) 171 : cluster [DBG] pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:37.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:37 smithi007 bash[25009]: cluster 2023-12-16T07:39:36.278756+0000 mgr.y (mgr.24367) 171 : cluster [DBG] pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:37.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:37 smithi007 bash[30436]: cluster 2023-12-16T07:39:36.278756+0000 mgr.y (mgr.24367) 171 : cluster [DBG] pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:39.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:39 smithi169 bash[25083]: cluster 2023-12-16T07:39:38.279496+0000 mgr.y (mgr.24367) 172 : cluster [DBG] pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:39.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:39 smithi007 bash[25009]: cluster 2023-12-16T07:39:38.279496+0000 mgr.y (mgr.24367) 172 : cluster [DBG] pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:39.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:39 smithi007 bash[30436]: cluster 2023-12-16T07:39:38.279496+0000 mgr.y (mgr.24367) 172 : cluster [DBG] pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:41.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:41 smithi169 bash[25083]: cluster 2023-12-16T07:39:40.280259+0000 mgr.y (mgr.24367) 173 : cluster [DBG] pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:41.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:41 smithi007 bash[25009]: cluster 2023-12-16T07:39:40.280259+0000 mgr.y (mgr.24367) 173 : cluster [DBG] pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:41.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:41 smithi007 bash[30436]: cluster 2023-12-16T07:39:40.280259+0000 mgr.y (mgr.24367) 173 : cluster [DBG] pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:42.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:39:42 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:39:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:39:43.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:43 smithi169 bash[25083]: cluster 2023-12-16T07:39:42.282004+0000 mgr.y (mgr.24367) 174 : cluster [DBG] pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:43.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:43 smithi007 bash[25009]: cluster 2023-12-16T07:39:42.282004+0000 mgr.y (mgr.24367) 174 : cluster [DBG] pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:43.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:43 smithi007 bash[30436]: cluster 2023-12-16T07:39:42.282004+0000 mgr.y (mgr.24367) 174 : cluster [DBG] pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:44.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:39:43 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:39:43] "GET /metrics HTTP/1.1" 200 214895 "" "Prometheus/2.33.4" 2023-12-16T07:39:44.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:44 smithi169 bash[25083]: audit 2023-12-16T07:39:43.631176+0000 mgr.y (mgr.24367) 175 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:44.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:44 smithi007 bash[25009]: audit 2023-12-16T07:39:43.631176+0000 mgr.y (mgr.24367) 175 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:44.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:44 smithi007 bash[30436]: audit 2023-12-16T07:39:43.631176+0000 mgr.y (mgr.24367) 175 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:45.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:45 smithi007 bash[25009]: cluster 2023-12-16T07:39:44.282764+0000 mgr.y (mgr.24367) 176 : cluster [DBG] pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:45.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:45 smithi007 bash[30436]: cluster 2023-12-16T07:39:44.282764+0000 mgr.y (mgr.24367) 176 : cluster [DBG] pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:45.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:45 smithi169 bash[25083]: cluster 2023-12-16T07:39:44.282764+0000 mgr.y (mgr.24367) 176 : cluster [DBG] pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:47.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:47 smithi169 bash[25083]: cluster 2023-12-16T07:39:46.284391+0000 mgr.y (mgr.24367) 177 : cluster [DBG] pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:47.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:47 smithi169 bash[25083]: audit 2023-12-16T07:39:46.481216+0000 mon.a (mon.0) 858 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:39:47.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:47 smithi169 bash[25083]: audit 2023-12-16T07:39:46.483218+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:39:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:47 smithi169 bash[25083]: audit 2023-12-16T07:39:46.777426+0000 mon.a (mon.0) 860 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:39:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:47 smithi169 bash[25083]: audit 2023-12-16T07:39:46.783453+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:39:47.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:47 smithi169 bash[25083]: audit 2023-12-16T07:39:47.092020+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:39:47.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:47 smithi007 bash[25009]: cluster 2023-12-16T07:39:46.284391+0000 mgr.y (mgr.24367) 177 : cluster [DBG] pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:47.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:47 smithi007 bash[25009]: audit 2023-12-16T07:39:46.481216+0000 mon.a (mon.0) 858 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:39:47.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:47 smithi007 bash[25009]: audit 2023-12-16T07:39:46.483218+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:39:47.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:47 smithi007 bash[25009]: audit 2023-12-16T07:39:46.777426+0000 mon.a (mon.0) 860 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:39:47.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:47 smithi007 bash[25009]: audit 2023-12-16T07:39:46.783453+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:39:47.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:47 smithi007 bash[25009]: audit 2023-12-16T07:39:47.092020+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:39:47.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:47 smithi007 bash[30436]: cluster 2023-12-16T07:39:46.284391+0000 mgr.y (mgr.24367) 177 : cluster [DBG] pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:47.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:47 smithi007 bash[30436]: audit 2023-12-16T07:39:46.481216+0000 mon.a (mon.0) 858 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:39:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:47 smithi007 bash[30436]: audit 2023-12-16T07:39:46.483218+0000 mon.a (mon.0) 859 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:39:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:47 smithi007 bash[30436]: audit 2023-12-16T07:39:46.777426+0000 mon.a (mon.0) 860 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:39:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:47 smithi007 bash[30436]: audit 2023-12-16T07:39:46.783453+0000 mon.a (mon.0) 861 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:39:47.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:47 smithi007 bash[30436]: audit 2023-12-16T07:39:47.092020+0000 mon.a (mon.0) 862 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:39:49.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:49 smithi169 bash[25083]: cluster 2023-12-16T07:39:48.285173+0000 mgr.y (mgr.24367) 178 : cluster [DBG] pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:49.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:49 smithi007 bash[25009]: cluster 2023-12-16T07:39:48.285173+0000 mgr.y (mgr.24367) 178 : cluster [DBG] pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:49 smithi007 bash[30436]: cluster 2023-12-16T07:39:48.285173+0000 mgr.y (mgr.24367) 178 : cluster [DBG] pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:51.688 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:51 smithi169 bash[25083]: cluster 2023-12-16T07:39:50.285996+0000 mgr.y (mgr.24367) 179 : cluster [DBG] pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:51.688 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:51 smithi169 bash[25083]: audit 2023-12-16T07:39:50.535618+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:39:51.688 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:51 smithi169 bash[25083]: audit 2023-12-16T07:39:50.546253+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:39:51.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:51 smithi007 bash[25009]: cluster 2023-12-16T07:39:50.285996+0000 mgr.y (mgr.24367) 179 : cluster [DBG] pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:51.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:51 smithi007 bash[25009]: audit 2023-12-16T07:39:50.535618+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:39:51.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:51 smithi007 bash[25009]: audit 2023-12-16T07:39:50.546253+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:39:51.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:51 smithi007 bash[30436]: cluster 2023-12-16T07:39:50.285996+0000 mgr.y (mgr.24367) 179 : cluster [DBG] pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:51.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:51 smithi007 bash[30436]: audit 2023-12-16T07:39:50.535618+0000 mon.a (mon.0) 863 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:39:51.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:51 smithi007 bash[30436]: audit 2023-12-16T07:39:50.546253+0000 mon.a (mon.0) 864 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:39:52.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:39:52 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:39:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:39:53.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:53 smithi169 bash[25083]: cluster 2023-12-16T07:39:52.293366+0000 mgr.y (mgr.24367) 180 : cluster [DBG] pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:53.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:53 smithi007 bash[25009]: cluster 2023-12-16T07:39:52.293366+0000 mgr.y (mgr.24367) 180 : cluster [DBG] pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:53.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:53 smithi007 bash[30436]: cluster 2023-12-16T07:39:52.293366+0000 mgr.y (mgr.24367) 180 : cluster [DBG] pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:54.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:39:53 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:39:53] "GET /metrics HTTP/1.1" 200 214881 "" "Prometheus/2.33.4" 2023-12-16T07:39:54.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:54 smithi169 bash[25083]: audit 2023-12-16T07:39:53.637516+0000 mgr.y (mgr.24367) 181 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:54.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:54 smithi007 bash[25009]: audit 2023-12-16T07:39:53.637516+0000 mgr.y (mgr.24367) 181 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:54.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:54 smithi007 bash[30436]: audit 2023-12-16T07:39:53.637516+0000 mgr.y (mgr.24367) 181 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:39:55.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:55 smithi007 bash[25009]: cluster 2023-12-16T07:39:54.294235+0000 mgr.y (mgr.24367) 182 : cluster [DBG] pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:55.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:55 smithi007 bash[30436]: cluster 2023-12-16T07:39:54.294235+0000 mgr.y (mgr.24367) 182 : cluster [DBG] pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:55 smithi169 bash[25083]: cluster 2023-12-16T07:39:54.294235+0000 mgr.y (mgr.24367) 182 : cluster [DBG] pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:57.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:57 smithi169 bash[25083]: cluster 2023-12-16T07:39:56.295990+0000 mgr.y (mgr.24367) 183 : cluster [DBG] pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:57.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:57 smithi007 bash[25009]: cluster 2023-12-16T07:39:56.295990+0000 mgr.y (mgr.24367) 183 : cluster [DBG] pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:57.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:57 smithi007 bash[30436]: cluster 2023-12-16T07:39:56.295990+0000 mgr.y (mgr.24367) 183 : cluster [DBG] pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:39:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:39:59 smithi169 bash[25083]: cluster 2023-12-16T07:39:58.296742+0000 mgr.y (mgr.24367) 184 : cluster [DBG] pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:59.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:39:59 smithi007 bash[25009]: cluster 2023-12-16T07:39:58.296742+0000 mgr.y (mgr.24367) 184 : cluster [DBG] pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:39:59.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:39:59 smithi007 bash[30436]: cluster 2023-12-16T07:39:58.296742+0000 mgr.y (mgr.24367) 184 : cluster [DBG] pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:00.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:00 smithi169 bash[25083]: cluster 2023-12-16T07:40:00.000213+0000 mon.a (mon.0) 865 : cluster [INF] overall HEALTH_OK 2023-12-16T07:40:00.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:00 smithi007 bash[25009]: cluster 2023-12-16T07:40:00.000213+0000 mon.a (mon.0) 865 : cluster [INF] overall HEALTH_OK 2023-12-16T07:40:00.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:00 smithi007 bash[30436]: cluster 2023-12-16T07:40:00.000213+0000 mon.a (mon.0) 865 : cluster [INF] overall HEALTH_OK 2023-12-16T07:40:01.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:01 smithi169 bash[25083]: cluster 2023-12-16T07:40:00.297507+0000 mgr.y (mgr.24367) 185 : cluster [DBG] pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:01.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:01 smithi007 bash[25009]: cluster 2023-12-16T07:40:00.297507+0000 mgr.y (mgr.24367) 185 : cluster [DBG] pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:01.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:01 smithi007 bash[30436]: cluster 2023-12-16T07:40:00.297507+0000 mgr.y (mgr.24367) 185 : cluster [DBG] pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:02.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:40:02 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:40:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:40:03.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:03 smithi169 bash[25083]: cluster 2023-12-16T07:40:02.299268+0000 mgr.y (mgr.24367) 186 : cluster [DBG] pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:03.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:03 smithi007 bash[25009]: cluster 2023-12-16T07:40:02.299268+0000 mgr.y (mgr.24367) 186 : cluster [DBG] pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:03.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:03 smithi007 bash[30436]: cluster 2023-12-16T07:40:02.299268+0000 mgr.y (mgr.24367) 186 : cluster [DBG] pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:04.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:40:03 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:40:03] "GET /metrics HTTP/1.1" 200 214881 "" "Prometheus/2.33.4" 2023-12-16T07:40:04.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:04 smithi169 bash[25083]: audit 2023-12-16T07:40:03.643344+0000 mgr.y (mgr.24367) 187 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:04.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:04 smithi007 bash[25009]: audit 2023-12-16T07:40:03.643344+0000 mgr.y (mgr.24367) 187 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:04.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:04 smithi007 bash[30436]: audit 2023-12-16T07:40:03.643344+0000 mgr.y (mgr.24367) 187 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:05.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:05 smithi007 bash[30436]: cluster 2023-12-16T07:40:04.300025+0000 mgr.y (mgr.24367) 188 : cluster [DBG] pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:05.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:05 smithi007 bash[25009]: cluster 2023-12-16T07:40:04.300025+0000 mgr.y (mgr.24367) 188 : cluster [DBG] pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:05.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:05 smithi169 bash[25083]: cluster 2023-12-16T07:40:04.300025+0000 mgr.y (mgr.24367) 188 : cluster [DBG] pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:07.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:07 smithi169 bash[25083]: cluster 2023-12-16T07:40:06.301821+0000 mgr.y (mgr.24367) 189 : cluster [DBG] pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:07.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:07 smithi007 bash[25009]: cluster 2023-12-16T07:40:06.301821+0000 mgr.y (mgr.24367) 189 : cluster [DBG] pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:07.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:07 smithi007 bash[30436]: cluster 2023-12-16T07:40:06.301821+0000 mgr.y (mgr.24367) 189 : cluster [DBG] pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:09.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:09 smithi169 bash[25083]: cluster 2023-12-16T07:40:08.302766+0000 mgr.y (mgr.24367) 190 : cluster [DBG] pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:09.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:09 smithi007 bash[30436]: cluster 2023-12-16T07:40:08.302766+0000 mgr.y (mgr.24367) 190 : cluster [DBG] pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:09.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:09 smithi007 bash[25009]: cluster 2023-12-16T07:40:08.302766+0000 mgr.y (mgr.24367) 190 : cluster [DBG] pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:11.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:11 smithi169 bash[25083]: cluster 2023-12-16T07:40:10.303637+0000 mgr.y (mgr.24367) 191 : cluster [DBG] pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:11.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:11 smithi007 bash[30436]: cluster 2023-12-16T07:40:10.303637+0000 mgr.y (mgr.24367) 191 : cluster [DBG] pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:11 smithi007 bash[25009]: cluster 2023-12-16T07:40:10.303637+0000 mgr.y (mgr.24367) 191 : cluster [DBG] pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:12.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:40:12 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:40:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:40:13.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:13 smithi007 bash[30436]: cluster 2023-12-16T07:40:12.305440+0000 mgr.y (mgr.24367) 192 : cluster [DBG] pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:13.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:13 smithi007 bash[25009]: cluster 2023-12-16T07:40:12.305440+0000 mgr.y (mgr.24367) 192 : cluster [DBG] pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:13.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:13 smithi169 bash[25083]: cluster 2023-12-16T07:40:12.305440+0000 mgr.y (mgr.24367) 192 : cluster [DBG] pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:14.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:40:13 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:40:13] "GET /metrics HTTP/1.1" 200 214901 "" "Prometheus/2.33.4" 2023-12-16T07:40:14.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:14 smithi007 bash[30436]: audit 2023-12-16T07:40:13.645006+0000 mgr.y (mgr.24367) 193 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:14.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:14 smithi007 bash[25009]: audit 2023-12-16T07:40:13.645006+0000 mgr.y (mgr.24367) 193 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:14.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:14 smithi169 bash[25083]: audit 2023-12-16T07:40:13.645006+0000 mgr.y (mgr.24367) 193 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:15.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:15 smithi007 bash[30436]: cluster 2023-12-16T07:40:14.306228+0000 mgr.y (mgr.24367) 194 : cluster [DBG] pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:15.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:15 smithi007 bash[25009]: cluster 2023-12-16T07:40:14.306228+0000 mgr.y (mgr.24367) 194 : cluster [DBG] pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:15 smithi169 bash[25083]: cluster 2023-12-16T07:40:14.306228+0000 mgr.y (mgr.24367) 194 : cluster [DBG] pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:17.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:17 smithi169 bash[25083]: cluster 2023-12-16T07:40:16.307922+0000 mgr.y (mgr.24367) 195 : cluster [DBG] pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:17.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:17 smithi007 bash[30436]: cluster 2023-12-16T07:40:16.307922+0000 mgr.y (mgr.24367) 195 : cluster [DBG] pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:17.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:17 smithi007 bash[25009]: cluster 2023-12-16T07:40:16.307922+0000 mgr.y (mgr.24367) 195 : cluster [DBG] pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:19.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:19 smithi169 bash[25083]: cluster 2023-12-16T07:40:18.308728+0000 mgr.y (mgr.24367) 196 : cluster [DBG] pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:19.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:19 smithi007 bash[25009]: cluster 2023-12-16T07:40:18.308728+0000 mgr.y (mgr.24367) 196 : cluster [DBG] pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:19.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:19 smithi007 bash[30436]: cluster 2023-12-16T07:40:18.308728+0000 mgr.y (mgr.24367) 196 : cluster [DBG] pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:21 smithi169 bash[25083]: cluster 2023-12-16T07:40:20.309407+0000 mgr.y (mgr.24367) 197 : cluster [DBG] pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:21.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:21 smithi007 bash[25009]: cluster 2023-12-16T07:40:20.309407+0000 mgr.y (mgr.24367) 197 : cluster [DBG] pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:21.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:21 smithi007 bash[30436]: cluster 2023-12-16T07:40:20.309407+0000 mgr.y (mgr.24367) 197 : cluster [DBG] pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:22.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:40:22 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:40:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:40:23.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:23 smithi007 bash[30436]: cluster 2023-12-16T07:40:22.311162+0000 mgr.y (mgr.24367) 198 : cluster [DBG] pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:23.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:23 smithi007 bash[25009]: cluster 2023-12-16T07:40:22.311162+0000 mgr.y (mgr.24367) 198 : cluster [DBG] pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:23.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:23 smithi169 bash[25083]: cluster 2023-12-16T07:40:22.311162+0000 mgr.y (mgr.24367) 198 : cluster [DBG] pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:24.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:40:23 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:40:23] "GET /metrics HTTP/1.1" 200 214907 "" "Prometheus/2.33.4" 2023-12-16T07:40:24.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:24 smithi007 bash[30436]: audit 2023-12-16T07:40:23.650530+0000 mgr.y (mgr.24367) 199 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:24.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:24 smithi007 bash[25009]: audit 2023-12-16T07:40:23.650530+0000 mgr.y (mgr.24367) 199 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:24.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:24 smithi169 bash[25083]: audit 2023-12-16T07:40:23.650530+0000 mgr.y (mgr.24367) 199 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:25.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:25 smithi007 bash[25009]: cluster 2023-12-16T07:40:24.311953+0000 mgr.y (mgr.24367) 200 : cluster [DBG] pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:25.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:25 smithi007 bash[30436]: cluster 2023-12-16T07:40:24.311953+0000 mgr.y (mgr.24367) 200 : cluster [DBG] pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:25 smithi169 bash[25083]: cluster 2023-12-16T07:40:24.311953+0000 mgr.y (mgr.24367) 200 : cluster [DBG] pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:27.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:27 smithi169 bash[25083]: cluster 2023-12-16T07:40:26.313628+0000 mgr.y (mgr.24367) 201 : cluster [DBG] pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:27.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:27 smithi007 bash[25009]: cluster 2023-12-16T07:40:26.313628+0000 mgr.y (mgr.24367) 201 : cluster [DBG] pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:27.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:27 smithi007 bash[30436]: cluster 2023-12-16T07:40:26.313628+0000 mgr.y (mgr.24367) 201 : cluster [DBG] pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:29.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:29 smithi169 bash[25083]: cluster 2023-12-16T07:40:28.314368+0000 mgr.y (mgr.24367) 202 : cluster [DBG] pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:29.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:29 smithi007 bash[25009]: cluster 2023-12-16T07:40:28.314368+0000 mgr.y (mgr.24367) 202 : cluster [DBG] pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:29.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:29 smithi007 bash[30436]: cluster 2023-12-16T07:40:28.314368+0000 mgr.y (mgr.24367) 202 : cluster [DBG] pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:31.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:31 smithi169 bash[25083]: cluster 2023-12-16T07:40:30.315122+0000 mgr.y (mgr.24367) 203 : cluster [DBG] pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:31.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:31 smithi007 bash[30436]: cluster 2023-12-16T07:40:30.315122+0000 mgr.y (mgr.24367) 203 : cluster [DBG] pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:31.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:31 smithi007 bash[25009]: cluster 2023-12-16T07:40:30.315122+0000 mgr.y (mgr.24367) 203 : cluster [DBG] pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:32.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:40:32 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:40:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:40:33.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:33 smithi007 bash[30436]: cluster 2023-12-16T07:40:32.316887+0000 mgr.y (mgr.24367) 204 : cluster [DBG] pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:33.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:33 smithi007 bash[25009]: cluster 2023-12-16T07:40:32.316887+0000 mgr.y (mgr.24367) 204 : cluster [DBG] pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:33.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:33 smithi169 bash[25083]: cluster 2023-12-16T07:40:32.316887+0000 mgr.y (mgr.24367) 204 : cluster [DBG] pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:34.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:40:33 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:40:33] "GET /metrics HTTP/1.1" 200 214907 "" "Prometheus/2.33.4" 2023-12-16T07:40:34.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:34 smithi007 bash[30436]: audit 2023-12-16T07:40:33.660670+0000 mgr.y (mgr.24367) 205 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:34.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:34 smithi007 bash[25009]: audit 2023-12-16T07:40:33.660670+0000 mgr.y (mgr.24367) 205 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:34 smithi169 bash[25083]: audit 2023-12-16T07:40:33.660670+0000 mgr.y (mgr.24367) 205 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:35.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:35 smithi007 bash[25009]: cluster 2023-12-16T07:40:34.317716+0000 mgr.y (mgr.24367) 206 : cluster [DBG] pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:35.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:35 smithi007 bash[30436]: cluster 2023-12-16T07:40:34.317716+0000 mgr.y (mgr.24367) 206 : cluster [DBG] pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:35 smithi169 bash[25083]: cluster 2023-12-16T07:40:34.317716+0000 mgr.y (mgr.24367) 206 : cluster [DBG] pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:37.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:37 smithi169 bash[25083]: cluster 2023-12-16T07:40:36.319437+0000 mgr.y (mgr.24367) 207 : cluster [DBG] pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:37.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:37 smithi007 bash[25009]: cluster 2023-12-16T07:40:36.319437+0000 mgr.y (mgr.24367) 207 : cluster [DBG] pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:37.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:37 smithi007 bash[30436]: cluster 2023-12-16T07:40:36.319437+0000 mgr.y (mgr.24367) 207 : cluster [DBG] pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:39.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:39 smithi169 bash[25083]: cluster 2023-12-16T07:40:38.320218+0000 mgr.y (mgr.24367) 208 : cluster [DBG] pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:39.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:39 smithi007 bash[30436]: cluster 2023-12-16T07:40:38.320218+0000 mgr.y (mgr.24367) 208 : cluster [DBG] pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:39.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:39 smithi007 bash[25009]: cluster 2023-12-16T07:40:38.320218+0000 mgr.y (mgr.24367) 208 : cluster [DBG] pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:41.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:41 smithi169 bash[25083]: cluster 2023-12-16T07:40:40.320987+0000 mgr.y (mgr.24367) 209 : cluster [DBG] pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:41.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:41 smithi007 bash[30436]: cluster 2023-12-16T07:40:40.320987+0000 mgr.y (mgr.24367) 209 : cluster [DBG] pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:41.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:41 smithi007 bash[25009]: cluster 2023-12-16T07:40:40.320987+0000 mgr.y (mgr.24367) 209 : cluster [DBG] pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:42.684 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:40:42 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:40:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:40:43.459 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:43 smithi007 bash[30436]: cluster 2023-12-16T07:40:42.322826+0000 mgr.y (mgr.24367) 210 : cluster [DBG] pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:43.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:43 smithi007 bash[25009]: cluster 2023-12-16T07:40:42.322826+0000 mgr.y (mgr.24367) 210 : cluster [DBG] pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:43.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:43 smithi169 bash[25083]: cluster 2023-12-16T07:40:42.322826+0000 mgr.y (mgr.24367) 210 : cluster [DBG] pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:44.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:40:43 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:40:43] "GET /metrics HTTP/1.1" 200 214901 "" "Prometheus/2.33.4" 2023-12-16T07:40:44.161 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2023-12-16T07:40:44.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:44 smithi007 bash[25009]: audit 2023-12-16T07:40:43.671150+0000 mgr.y (mgr.24367) 211 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:44.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:44 smithi007 bash[30436]: audit 2023-12-16T07:40:43.671150+0000 mgr.y (mgr.24367) 211 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:44.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:44 smithi169 bash[25083]: audit 2023-12-16T07:40:43.671150+0000 mgr.y (mgr.24367) 211 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:45.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:45 smithi007 bash[25009]: cluster 2023-12-16T07:40:44.323523+0000 mgr.y (mgr.24367) 212 : cluster [DBG] pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:45.460 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:45 smithi007 bash[30436]: cluster 2023-12-16T07:40:44.323523+0000 mgr.y (mgr.24367) 212 : cluster [DBG] pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:45.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:45 smithi169 bash[25083]: cluster 2023-12-16T07:40:44.323523+0000 mgr.y (mgr.24367) 212 : cluster [DBG] pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:47.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:47 smithi169 bash[25083]: cluster 2023-12-16T07:40:46.324734+0000 mgr.y (mgr.24367) 213 : cluster [DBG] pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:47.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:47 smithi169 bash[25083]: audit 2023-12-16T07:40:47.103841+0000 mon.a (mon.0) 866 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:40:47.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:47 smithi169 bash[25083]: audit 2023-12-16T07:40:47.105884+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:40:47.758 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:47 smithi007 bash[25009]: cluster 2023-12-16T07:40:46.324734+0000 mgr.y (mgr.24367) 213 : cluster [DBG] pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:47.758 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:47 smithi007 bash[25009]: audit 2023-12-16T07:40:47.103841+0000 mon.a (mon.0) 866 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:40:47.758 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:47 smithi007 bash[25009]: audit 2023-12-16T07:40:47.105884+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:40:47.758 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:47 smithi007 bash[30436]: cluster 2023-12-16T07:40:46.324734+0000 mgr.y (mgr.24367) 213 : cluster [DBG] pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:47.758 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:47 smithi007 bash[30436]: audit 2023-12-16T07:40:47.103841+0000 mon.a (mon.0) 866 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:40:47.759 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:47 smithi007 bash[30436]: audit 2023-12-16T07:40:47.105884+0000 mon.a (mon.0) 867 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:40:48.500 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2023-12-16T07:40:48.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:48 smithi169 bash[25083]: audit 2023-12-16T07:40:47.423656+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:40:48.699 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:48 smithi007 bash[30436]: audit 2023-12-16T07:40:47.423656+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:40:48.700 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:48 smithi007 bash[25009]: audit 2023-12-16T07:40:47.423656+0000 mon.a (mon.0) 868 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:40:49.722 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:49 smithi007 bash[30436]: cluster 2023-12-16T07:40:48.325473+0000 mgr.y (mgr.24367) 214 : cluster [DBG] pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:49.722 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:49 smithi007 bash[25009]: cluster 2023-12-16T07:40:48.325473+0000 mgr.y (mgr.24367) 214 : cluster [DBG] pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:49.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:49 smithi169 bash[25083]: cluster 2023-12-16T07:40:48.325473+0000 mgr.y (mgr.24367) 214 : cluster [DBG] pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:51.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:51 smithi007 bash[25009]: cluster 2023-12-16T07:40:50.326308+0000 mgr.y (mgr.24367) 215 : cluster [DBG] pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:51.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:51 smithi007 bash[25009]: audit 2023-12-16T07:40:50.541355+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:40:51.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:51 smithi007 bash[25009]: audit 2023-12-16T07:40:50.552149+0000 mon.a (mon.0) 870 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:40:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:51 smithi007 bash[30436]: cluster 2023-12-16T07:40:50.326308+0000 mgr.y (mgr.24367) 215 : cluster [DBG] pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:51 smithi007 bash[30436]: audit 2023-12-16T07:40:50.541355+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:40:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:51 smithi007 bash[30436]: audit 2023-12-16T07:40:50.552149+0000 mon.a (mon.0) 870 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:40:51.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:51 smithi169 bash[25083]: cluster 2023-12-16T07:40:50.326308+0000 mgr.y (mgr.24367) 215 : cluster [DBG] pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:51.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:51 smithi169 bash[25083]: audit 2023-12-16T07:40:50.541355+0000 mon.a (mon.0) 869 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:40:51.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:51 smithi169 bash[25083]: audit 2023-12-16T07:40:50.552149+0000 mon.a (mon.0) 870 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:40:52.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:40:52 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:40:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:40:52.826 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph config set global log_to_journald false --force' 2023-12-16T07:40:53.199 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:53 smithi007 bash[30436]: cluster 2023-12-16T07:40:52.328138+0000 mgr.y (mgr.24367) 216 : cluster [DBG] pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:53.199 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:53 smithi007 bash[25009]: cluster 2023-12-16T07:40:52.328138+0000 mgr.y (mgr.24367) 216 : cluster [DBG] pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:53.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:53 smithi169 bash[25083]: cluster 2023-12-16T07:40:52.328138+0000 mgr.y (mgr.24367) 216 : cluster [DBG] pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:54.021 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:40:53 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:40:53] "GET /metrics HTTP/1.1" 200 214896 "" "Prometheus/2.33.4" 2023-12-16T07:40:54.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:54 smithi007 bash[25009]: audit 2023-12-16T07:40:53.678655+0000 mgr.y (mgr.24367) 217 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:54.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:54 smithi007 bash[30436]: audit 2023-12-16T07:40:53.678655+0000 mgr.y (mgr.24367) 217 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:54.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:54 smithi169 bash[25083]: audit 2023-12-16T07:40:53.678655+0000 mgr.y (mgr.24367) 217 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:40:55.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:55 smithi007 bash[30436]: cluster 2023-12-16T07:40:54.329045+0000 mgr.y (mgr.24367) 218 : cluster [DBG] pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:55.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:55 smithi007 bash[25009]: cluster 2023-12-16T07:40:54.329045+0000 mgr.y (mgr.24367) 218 : cluster [DBG] pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:55 smithi169 bash[25083]: cluster 2023-12-16T07:40:54.329045+0000 mgr.y (mgr.24367) 218 : cluster [DBG] pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:56.977 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2023-12-16T07:40:57.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:57 smithi007 bash[25009]: cluster 2023-12-16T07:40:56.330949+0000 mgr.y (mgr.24367) 219 : cluster [DBG] pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:57.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:57 smithi007 bash[30436]: cluster 2023-12-16T07:40:56.330949+0000 mgr.y (mgr.24367) 219 : cluster [DBG] pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:57.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:57 smithi169 bash[25083]: cluster 2023-12-16T07:40:56.330949+0000 mgr.y (mgr.24367) 219 : cluster [DBG] pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:40:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:40:59 smithi169 bash[25083]: cluster 2023-12-16T07:40:58.331825+0000 mgr.y (mgr.24367) 220 : cluster [DBG] pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:59.739 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:40:59 smithi007 bash[30436]: cluster 2023-12-16T07:40:58.331825+0000 mgr.y (mgr.24367) 220 : cluster [DBG] pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:40:59.740 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:40:59 smithi007 bash[25009]: cluster 2023-12-16T07:40:58.331825+0000 mgr.y (mgr.24367) 220 : cluster [DBG] pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:00.166 INFO:teuthology.orchestra.run.smithi007.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T07:41:01.101 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-12-16T07:41:01.112 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2023-12-16T07:41:01.112 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2023-12-16T07:41:01.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:01 smithi169 bash[25083]: audit 2023-12-16T07:41:00.157443+0000 mgr.y (mgr.24367) 221 : audit [DBG] from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:01.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:01 smithi169 bash[25083]: cephadm 2023-12-16T07:41:00.159259+0000 mgr.y (mgr.24367) 222 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T07:41:01.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:01 smithi169 bash[25083]: audit 2023-12-16T07:41:00.164604+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:41:01.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:01 smithi169 bash[25083]: audit 2023-12-16T07:41:00.181077+0000 mon.a (mon.0) 872 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:41:01.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:01 smithi169 bash[25083]: audit 2023-12-16T07:41:00.182758+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:41:01.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:01 smithi169 bash[25083]: audit 2023-12-16T07:41:00.191813+0000 mon.a (mon.0) 874 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:41:01.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:01 smithi169 bash[25083]: cephadm 2023-12-16T07:41:00.199800+0000 mgr.y (mgr.24367) 223 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T07:41:01.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:01 smithi169 bash[25083]: cluster 2023-12-16T07:41:00.332582+0000 mgr.y (mgr.24367) 224 : cluster [DBG] pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:01.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[30436]: audit 2023-12-16T07:41:00.157443+0000 mgr.y (mgr.24367) 221 : audit [DBG] from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:01.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[30436]: cephadm 2023-12-16T07:41:00.159259+0000 mgr.y (mgr.24367) 222 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T07:41:01.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[30436]: audit 2023-12-16T07:41:00.164604+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:41:01.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[30436]: audit 2023-12-16T07:41:00.181077+0000 mon.a (mon.0) 872 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:41:01.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[30436]: audit 2023-12-16T07:41:00.182758+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:41:01.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[30436]: audit 2023-12-16T07:41:00.191813+0000 mon.a (mon.0) 874 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:41:01.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[30436]: cephadm 2023-12-16T07:41:00.199800+0000 mgr.y (mgr.24367) 223 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T07:41:01.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[30436]: cluster 2023-12-16T07:41:00.332582+0000 mgr.y (mgr.24367) 224 : cluster [DBG] pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:01.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[25009]: audit 2023-12-16T07:41:00.157443+0000 mgr.y (mgr.24367) 221 : audit [DBG] from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:01.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[25009]: cephadm 2023-12-16T07:41:00.159259+0000 mgr.y (mgr.24367) 222 : cephadm [INF] Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T07:41:01.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[25009]: audit 2023-12-16T07:41:00.164604+0000 mon.a (mon.0) 871 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:41:01.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[25009]: audit 2023-12-16T07:41:00.181077+0000 mon.a (mon.0) 872 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:41:01.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[25009]: audit 2023-12-16T07:41:00.182758+0000 mon.a (mon.0) 873 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:41:01.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[25009]: audit 2023-12-16T07:41:00.191813+0000 mon.a (mon.0) 874 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:41:01.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[25009]: cephadm 2023-12-16T07:41:00.199800+0000 mgr.y (mgr.24367) 223 : cephadm [INF] Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678 2023-12-16T07:41:01.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:01 smithi007 bash[25009]: cluster 2023-12-16T07:41:00.332582+0000 mgr.y (mgr.24367) 224 : cluster [DBG] pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:02.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:41:02 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:41:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:41:03.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:03 smithi007 bash[30436]: cluster 2023-12-16T07:41:02.333904+0000 mgr.y (mgr.24367) 225 : cluster [DBG] pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:03.321 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:03 smithi007 bash[25009]: cluster 2023-12-16T07:41:02.333904+0000 mgr.y (mgr.24367) 225 : cluster [DBG] pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:03.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:03 smithi169 bash[25083]: cluster 2023-12-16T07:41:02.333904+0000 mgr.y (mgr.24367) 225 : cluster [DBG] pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:04.009 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:41:03 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:41:03] "GET /metrics HTTP/1.1" 200 214896 "" "Prometheus/2.33.4" 2023-12-16T07:41:04.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:04 smithi007 bash[25009]: audit 2023-12-16T07:41:03.687037+0000 mgr.y (mgr.24367) 226 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:04.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:04 smithi007 bash[30436]: audit 2023-12-16T07:41:03.687037+0000 mgr.y (mgr.24367) 226 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:04.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:04 smithi169 bash[25083]: audit 2023-12-16T07:41:03.687037+0000 mgr.y (mgr.24367) 226 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:05.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:05 smithi007 bash[25009]: cluster 2023-12-16T07:41:04.334503+0000 mgr.y (mgr.24367) 227 : cluster [DBG] pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:05.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:05 smithi007 bash[30436]: cluster 2023-12-16T07:41:04.334503+0000 mgr.y (mgr.24367) 227 : cluster [DBG] pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:05.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:05 smithi169 bash[25083]: cluster 2023-12-16T07:41:04.334503+0000 mgr.y (mgr.24367) 227 : cluster [DBG] pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:05.998 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:41:06.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:06 smithi007 bash[25009]: audit 2023-12-16T07:41:05.981409+0000 mgr.y (mgr.24367) 228 : audit [DBG] from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:06.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:06 smithi007 bash[30436]: audit 2023-12-16T07:41:05.981409+0000 mgr.y (mgr.24367) 228 : audit [DBG] from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:06 smithi169 bash[25083]: audit 2023-12-16T07:41:05.981409+0000 mgr.y (mgr.24367) 228 : audit [DBG] from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:06.906 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:41:06.906 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (3m) 2m ago 4m 16.5M - ba2b418f427c 128c156c0190 2023-12-16T07:41:06.906 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (4m) 2m ago 4m 46.0M - 8.3.5 dad864ee21e9 25b9cf47977e 2023-12-16T07:41:06.906 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (2m) 2m ago 2m 66.0M - 3.5 e1d6a67b021e ca15814d070f 2023-12-16T07:41:06.906 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443 running (10m) 2m ago 10m 405M - 17.2.0 e1d6a67b021e ff78f5dd3700 2023-12-16T07:41:06.906 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:9283 running (12m) 2m ago 12m 452M - 17.2.0 e1d6a67b021e 5219d4698403 2023-12-16T07:41:06.906 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (12m) 2m ago 12m 54.3M 2048M 17.2.0 e1d6a67b021e c0468ecbd44e 2023-12-16T07:41:06.906 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (11m) 2m ago 11m 40.0M 2048M 17.2.0 e1d6a67b021e b47a960c6102 2023-12-16T07:41:06.906 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (11m) 2m ago 11m 42.3M 2048M 17.2.0 e1d6a67b021e c659b986f201 2023-12-16T07:41:06.906 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (4m) 2m ago 4m 11.6M - 1dbe0e931976 54a71535c2b4 2023-12-16T07:41:06.906 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (4m) 2m ago 4m 13.0M - 1dbe0e931976 b050faea525d 2023-12-16T07:41:06.906 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (10m) 2m ago 10m 54.1M 4062M 17.2.0 e1d6a67b021e dbc4d9538238 2023-12-16T07:41:06.907 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (9m) 2m ago 9m 54.6M 4062M 17.2.0 e1d6a67b021e 14637e0f389c 2023-12-16T07:41:06.907 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (8m) 2m ago 8m 53.2M 4062M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:41:06.907 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (8m) 2m ago 8m 50.9M 4062M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:41:06.907 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (7m) 2m ago 7m 51.2M 4317M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:41:06.907 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (7m) 2m ago 7m 50.0M 4317M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:41:06.907 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (6m) 2m ago 6m 47.9M 4317M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:41:06.907 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (5m) 2m ago 5m 50.0M 4317M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:41:06.907 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (3m) 2m ago 4m 49.4M - 514e6a882f6e 9351c4c7d0ba 2023-12-16T07:41:06.907 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (2m) 2m ago 2m 83.5M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:41:06.907 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (2m) 2m ago 2m 83.1M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:41:06.907 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (2m) 2m ago 2m 83.5M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:41:06.907 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (2m) 2m ago 2m 82.8M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:41:07.435 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:41:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:41:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2023-12-16T07:41:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:41:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:41:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2023-12-16T07:41:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:41:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:41:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2023-12-16T07:41:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:41:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2023-12-16T07:41:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:41:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:41:07.436 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:41:07.437 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:41:07.437 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2023-12-16T07:41:07.437 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:41:07.437 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:41:07.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:07 smithi007 bash[25009]: cluster 2023-12-16T07:41:06.335977+0000 mgr.y (mgr.24367) 229 : cluster [DBG] pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:07.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:07 smithi007 bash[25009]: audit 2023-12-16T07:41:06.447447+0000 mgr.y (mgr.24367) 230 : audit [DBG] from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:07.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:07 smithi007 bash[25009]: audit 2023-12-16T07:41:06.880400+0000 mgr.y (mgr.24367) 231 : audit [DBG] from='client.24997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:07.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:07 smithi007 bash[30436]: cluster 2023-12-16T07:41:06.335977+0000 mgr.y (mgr.24367) 229 : cluster [DBG] pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:07.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:07 smithi007 bash[30436]: audit 2023-12-16T07:41:06.447447+0000 mgr.y (mgr.24367) 230 : audit [DBG] from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:07.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:07 smithi007 bash[30436]: audit 2023-12-16T07:41:06.880400+0000 mgr.y (mgr.24367) 231 : audit [DBG] from='client.24997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:07.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:07 smithi169 bash[25083]: cluster 2023-12-16T07:41:06.335977+0000 mgr.y (mgr.24367) 229 : cluster [DBG] pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:07.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:07 smithi169 bash[25083]: audit 2023-12-16T07:41:06.447447+0000 mgr.y (mgr.24367) 230 : audit [DBG] from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:07.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:07 smithi169 bash[25083]: audit 2023-12-16T07:41:06.880400+0000 mgr.y (mgr.24367) 231 : audit [DBG] from='client.24997 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:07.850 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:41:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:41:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:41:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [], 2023-12-16T07:41:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "", 2023-12-16T07:41:07.851 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678 image" 2023-12-16T07:41:07.851 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:41:08.310 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:41:08.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:08 smithi007 bash[30436]: audit 2023-12-16T07:41:07.434047+0000 mon.a (mon.0) 875 : audit [DBG] from='client.? 172.21.15.7:0/968977552' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:41:08.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:08 smithi007 bash[30436]: audit 2023-12-16T07:41:07.848791+0000 mgr.y (mgr.24367) 232 : audit [DBG] from='client.25009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:08.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:08 smithi007 bash[25009]: audit 2023-12-16T07:41:07.434047+0000 mon.a (mon.0) 875 : audit [DBG] from='client.? 172.21.15.7:0/968977552' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:41:08.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:08 smithi007 bash[25009]: audit 2023-12-16T07:41:07.848791+0000 mgr.y (mgr.24367) 232 : audit [DBG] from='client.25009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:08 smithi169 bash[25083]: audit 2023-12-16T07:41:07.434047+0000 mon.a (mon.0) 875 : audit [DBG] from='client.? 172.21.15.7:0/968977552' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:41:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:08 smithi169 bash[25083]: audit 2023-12-16T07:41:07.848791+0000 mgr.y (mgr.24367) 232 : audit [DBG] from='client.25009 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:09.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:09 smithi007 bash[25009]: audit 2023-12-16T07:41:08.304661+0000 mon.a (mon.0) 876 : audit [DBG] from='client.? 172.21.15.7:0/1117765338' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:41:09.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:09 smithi007 bash[25009]: cluster 2023-12-16T07:41:08.336549+0000 mgr.y (mgr.24367) 233 : cluster [DBG] pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:09.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:09 smithi007 bash[30436]: audit 2023-12-16T07:41:08.304661+0000 mon.a (mon.0) 876 : audit [DBG] from='client.? 172.21.15.7:0/1117765338' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:41:09.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:09 smithi007 bash[30436]: cluster 2023-12-16T07:41:08.336549+0000 mgr.y (mgr.24367) 233 : cluster [DBG] pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:09.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:09 smithi169 bash[25083]: audit 2023-12-16T07:41:08.304661+0000 mon.a (mon.0) 876 : audit [DBG] from='client.? 172.21.15.7:0/1117765338' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:41:09.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:09 smithi169 bash[25083]: cluster 2023-12-16T07:41:08.336549+0000 mgr.y (mgr.24367) 233 : cluster [DBG] pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:11.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:11 smithi169 bash[25083]: cluster 2023-12-16T07:41:10.337272+0000 mgr.y (mgr.24367) 234 : cluster [DBG] pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:11.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:11 smithi007 bash[30436]: cluster 2023-12-16T07:41:10.337272+0000 mgr.y (mgr.24367) 234 : cluster [DBG] pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:11 smithi007 bash[25009]: cluster 2023-12-16T07:41:10.337272+0000 mgr.y (mgr.24367) 234 : cluster [DBG] pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:12.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:41:12 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:41:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:41:13.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:13 smithi007 bash[25009]: cluster 2023-12-16T07:41:12.338631+0000 mgr.y (mgr.24367) 235 : cluster [DBG] pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:13.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:13 smithi007 bash[30436]: cluster 2023-12-16T07:41:12.338631+0000 mgr.y (mgr.24367) 235 : cluster [DBG] pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:13.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:13 smithi169 bash[25083]: cluster 2023-12-16T07:41:12.338631+0000 mgr.y (mgr.24367) 235 : cluster [DBG] pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:14.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:41:13 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:41:13] "GET /metrics HTTP/1.1" 200 214865 "" "Prometheus/2.33.4" 2023-12-16T07:41:14.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:14 smithi007 bash[30436]: audit 2023-12-16T07:41:13.697557+0000 mgr.y (mgr.24367) 236 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:14.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:14 smithi007 bash[25009]: audit 2023-12-16T07:41:13.697557+0000 mgr.y (mgr.24367) 236 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:14.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:14 smithi169 bash[25083]: audit 2023-12-16T07:41:13.697557+0000 mgr.y (mgr.24367) 236 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:15.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:15 smithi007 bash[30436]: cluster 2023-12-16T07:41:14.340061+0000 mgr.y (mgr.24367) 237 : cluster [DBG] pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:15.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:15 smithi007 bash[25009]: cluster 2023-12-16T07:41:14.340061+0000 mgr.y (mgr.24367) 237 : cluster [DBG] pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:15 smithi169 bash[25083]: cluster 2023-12-16T07:41:14.340061+0000 mgr.y (mgr.24367) 237 : cluster [DBG] pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:17.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:17 smithi169 bash[25083]: cluster 2023-12-16T07:41:16.341578+0000 mgr.y (mgr.24367) 238 : cluster [DBG] pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:17.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:17 smithi007 bash[30436]: cluster 2023-12-16T07:41:16.341578+0000 mgr.y (mgr.24367) 238 : cluster [DBG] pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:17.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:17 smithi007 bash[25009]: cluster 2023-12-16T07:41:16.341578+0000 mgr.y (mgr.24367) 238 : cluster [DBG] pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:19.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:19 smithi169 bash[25083]: cluster 2023-12-16T07:41:18.342285+0000 mgr.y (mgr.24367) 239 : cluster [DBG] pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:19.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:19 smithi007 bash[30436]: cluster 2023-12-16T07:41:18.342285+0000 mgr.y (mgr.24367) 239 : cluster [DBG] pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:19.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:19 smithi007 bash[25009]: cluster 2023-12-16T07:41:18.342285+0000 mgr.y (mgr.24367) 239 : cluster [DBG] pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:21 smithi169 bash[25083]: cluster 2023-12-16T07:41:20.342770+0000 mgr.y (mgr.24367) 240 : cluster [DBG] pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:21.816 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:21 smithi007 bash[25009]: cluster 2023-12-16T07:41:20.342770+0000 mgr.y (mgr.24367) 240 : cluster [DBG] pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:21.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:21 smithi007 bash[30436]: cluster 2023-12-16T07:41:20.342770+0000 mgr.y (mgr.24367) 240 : cluster [DBG] pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:22.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:41:22 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:41:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:41:23.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:23 smithi007 bash[30436]: cluster 2023-12-16T07:41:22.344680+0000 mgr.y (mgr.24367) 241 : cluster [DBG] pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:23.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:23 smithi007 bash[25009]: cluster 2023-12-16T07:41:22.344680+0000 mgr.y (mgr.24367) 241 : cluster [DBG] pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:23.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:23 smithi169 bash[25083]: cluster 2023-12-16T07:41:22.344680+0000 mgr.y (mgr.24367) 241 : cluster [DBG] pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:24.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:41:23 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:41:23] "GET /metrics HTTP/1.1" 200 214844 "" "Prometheus/2.33.4" 2023-12-16T07:41:24.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:24 smithi007 bash[25009]: audit 2023-12-16T07:41:23.708506+0000 mgr.y (mgr.24367) 242 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:24.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:24 smithi007 bash[30436]: audit 2023-12-16T07:41:23.708506+0000 mgr.y (mgr.24367) 242 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:24.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:24 smithi169 bash[25083]: audit 2023-12-16T07:41:23.708506+0000 mgr.y (mgr.24367) 242 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:25.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:25 smithi007 bash[25009]: cluster 2023-12-16T07:41:24.345415+0000 mgr.y (mgr.24367) 243 : cluster [DBG] pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:25.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:25 smithi007 bash[30436]: cluster 2023-12-16T07:41:24.345415+0000 mgr.y (mgr.24367) 243 : cluster [DBG] pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:25 smithi169 bash[25083]: cluster 2023-12-16T07:41:24.345415+0000 mgr.y (mgr.24367) 243 : cluster [DBG] pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:27.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:27 smithi169 bash[25083]: cluster 2023-12-16T07:41:26.346862+0000 mgr.y (mgr.24367) 244 : cluster [DBG] pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:27.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:27 smithi007 bash[30436]: cluster 2023-12-16T07:41:26.346862+0000 mgr.y (mgr.24367) 244 : cluster [DBG] pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:27.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:27 smithi007 bash[25009]: cluster 2023-12-16T07:41:26.346862+0000 mgr.y (mgr.24367) 244 : cluster [DBG] pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:29.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:29 smithi169 bash[25083]: cluster 2023-12-16T07:41:28.347408+0000 mgr.y (mgr.24367) 245 : cluster [DBG] pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:29.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:29 smithi007 bash[30436]: cluster 2023-12-16T07:41:28.347408+0000 mgr.y (mgr.24367) 245 : cluster [DBG] pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:29.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:29 smithi007 bash[25009]: cluster 2023-12-16T07:41:28.347408+0000 mgr.y (mgr.24367) 245 : cluster [DBG] pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:31.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:31 smithi169 bash[25083]: cluster 2023-12-16T07:41:30.347983+0000 mgr.y (mgr.24367) 246 : cluster [DBG] pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:31.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:31 smithi007 bash[30436]: cluster 2023-12-16T07:41:30.347983+0000 mgr.y (mgr.24367) 246 : cluster [DBG] pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:31.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:31 smithi007 bash[25009]: cluster 2023-12-16T07:41:30.347983+0000 mgr.y (mgr.24367) 246 : cluster [DBG] pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:32.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:41:32 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:41:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:41:33.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:33 smithi007 bash[30436]: cluster 2023-12-16T07:41:32.349737+0000 mgr.y (mgr.24367) 247 : cluster [DBG] pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:33.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:33 smithi007 bash[25009]: cluster 2023-12-16T07:41:32.349737+0000 mgr.y (mgr.24367) 247 : cluster [DBG] pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:33.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:33 smithi169 bash[25083]: cluster 2023-12-16T07:41:32.349737+0000 mgr.y (mgr.24367) 247 : cluster [DBG] pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:34.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:41:33 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:41:33] "GET /metrics HTTP/1.1" 200 214844 "" "Prometheus/2.33.4" 2023-12-16T07:41:34.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:34 smithi007 bash[30436]: audit 2023-12-16T07:41:33.719179+0000 mgr.y (mgr.24367) 248 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:34.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:34 smithi007 bash[25009]: audit 2023-12-16T07:41:33.719179+0000 mgr.y (mgr.24367) 248 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:34 smithi169 bash[25083]: audit 2023-12-16T07:41:33.719179+0000 mgr.y (mgr.24367) 248 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:35.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:35 smithi007 bash[30436]: cluster 2023-12-16T07:41:34.350487+0000 mgr.y (mgr.24367) 249 : cluster [DBG] pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:35.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:35 smithi007 bash[25009]: cluster 2023-12-16T07:41:34.350487+0000 mgr.y (mgr.24367) 249 : cluster [DBG] pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:35 smithi169 bash[25083]: cluster 2023-12-16T07:41:34.350487+0000 mgr.y (mgr.24367) 249 : cluster [DBG] pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:37.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:37 smithi169 bash[25083]: cluster 2023-12-16T07:41:36.352468+0000 mgr.y (mgr.24367) 250 : cluster [DBG] pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:37.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:37 smithi007 bash[30436]: cluster 2023-12-16T07:41:36.352468+0000 mgr.y (mgr.24367) 250 : cluster [DBG] pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:37.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:37 smithi007 bash[25009]: cluster 2023-12-16T07:41:36.352468+0000 mgr.y (mgr.24367) 250 : cluster [DBG] pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:38.698 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:41:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:41:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (4m) 2m ago 5m 16.5M - ba2b418f427c 128c156c0190 2023-12-16T07:41:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (4m) 3m ago 4m 46.0M - 8.3.5 dad864ee21e9 25b9cf47977e 2023-12-16T07:41:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (2m) 2m ago 2m 66.0M - 3.5 e1d6a67b021e ca15814d070f 2023-12-16T07:41:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443 running (11m) 3m ago 11m 405M - 17.2.0 e1d6a67b021e ff78f5dd3700 2023-12-16T07:41:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:9283 running (13m) 2m ago 13m 452M - 17.2.0 e1d6a67b021e 5219d4698403 2023-12-16T07:41:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (13m) 2m ago 13m 54.3M 2048M 17.2.0 e1d6a67b021e c0468ecbd44e 2023-12-16T07:41:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (11m) 3m ago 11m 40.0M 2048M 17.2.0 e1d6a67b021e b47a960c6102 2023-12-16T07:41:39.425 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (11m) 2m ago 11m 42.3M 2048M 17.2.0 e1d6a67b021e c659b986f201 2023-12-16T07:41:39.425 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (5m) 2m ago 5m 11.6M - 1dbe0e931976 54a71535c2b4 2023-12-16T07:41:39.425 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (5m) 3m ago 5m 13.0M - 1dbe0e931976 b050faea525d 2023-12-16T07:41:39.425 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (10m) 2m ago 10m 54.1M 4062M 17.2.0 e1d6a67b021e dbc4d9538238 2023-12-16T07:41:39.425 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (10m) 2m ago 10m 54.6M 4062M 17.2.0 e1d6a67b021e 14637e0f389c 2023-12-16T07:41:39.425 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (9m) 2m ago 9m 53.2M 4062M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:41:39.425 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (8m) 2m ago 8m 50.9M 4062M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:41:39.425 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (8m) 3m ago 8m 51.2M 4317M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:41:39.425 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (7m) 3m ago 7m 50.0M 4317M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:41:39.425 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (7m) 3m ago 7m 47.9M 4317M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:41:39.425 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (6m) 3m ago 6m 50.0M 4317M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:41:39.425 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (4m) 3m ago 5m 49.4M - 514e6a882f6e 9351c4c7d0ba 2023-12-16T07:41:39.425 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (3m) 2m ago 3m 83.5M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:41:39.425 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (3m) 3m ago 3m 83.1M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:41:39.426 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (3m) 2m ago 3m 83.5M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:41:39.426 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (3m) 3m ago 3m 82.8M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:41:39.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:39 smithi169 bash[25083]: cluster 2023-12-16T07:41:38.353102+0000 mgr.y (mgr.24367) 251 : cluster [DBG] pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:39.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:39 smithi169 bash[25083]: audit 2023-12-16T07:41:38.680782+0000 mgr.y (mgr.24367) 252 : audit [DBG] from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:39.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:39 smithi169 bash[25083]: audit 2023-12-16T07:41:39.047532+0000 mgr.y (mgr.24367) 253 : audit [DBG] from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:39.816 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:39 smithi007 bash[30436]: cluster 2023-12-16T07:41:38.353102+0000 mgr.y (mgr.24367) 251 : cluster [DBG] pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:39.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:39 smithi007 bash[30436]: audit 2023-12-16T07:41:38.680782+0000 mgr.y (mgr.24367) 252 : audit [DBG] from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:39.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:39 smithi007 bash[30436]: audit 2023-12-16T07:41:39.047532+0000 mgr.y (mgr.24367) 253 : audit [DBG] from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:39.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:39 smithi007 bash[25009]: cluster 2023-12-16T07:41:38.353102+0000 mgr.y (mgr.24367) 251 : cluster [DBG] pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:39.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:39 smithi007 bash[25009]: audit 2023-12-16T07:41:38.680782+0000 mgr.y (mgr.24367) 252 : audit [DBG] from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:39.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:39 smithi007 bash[25009]: audit 2023-12-16T07:41:39.047532+0000 mgr.y (mgr.24367) 253 : audit [DBG] from='client.25027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:39.850 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:41:39.850 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:41:39.850 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2023-12-16T07:41:39.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:41:39.850 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:41:39.850 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2023-12-16T07:41:39.850 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:41:39.851 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:41:39.851 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2023-12-16T07:41:39.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:41:39.851 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2023-12-16T07:41:39.851 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:41:39.851 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:41:39.851 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:41:39.851 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:41:39.851 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2023-12-16T07:41:39.851 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:41:39.851 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:41:40.209 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:41:40.209 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:41:40.210 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:41:40.210 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [], 2023-12-16T07:41:40.210 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "", 2023-12-16T07:41:40.210 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678 image" 2023-12-16T07:41:40.210 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:41:40.621 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:41:40.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:40 smithi169 bash[25083]: audit 2023-12-16T07:41:39.398741+0000 mgr.y (mgr.24367) 254 : audit [DBG] from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:40.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:40 smithi169 bash[25083]: audit 2023-12-16T07:41:39.847992+0000 mon.a (mon.0) 877 : audit [DBG] from='client.? 172.21.15.7:0/3480080907' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:41:40.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:40 smithi007 bash[25009]: audit 2023-12-16T07:41:39.398741+0000 mgr.y (mgr.24367) 254 : audit [DBG] from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:40.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:40 smithi007 bash[25009]: audit 2023-12-16T07:41:39.847992+0000 mon.a (mon.0) 877 : audit [DBG] from='client.? 172.21.15.7:0/3480080907' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:41:40.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:40 smithi007 bash[30436]: audit 2023-12-16T07:41:39.398741+0000 mgr.y (mgr.24367) 254 : audit [DBG] from='client.25033 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:40.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:40 smithi007 bash[30436]: audit 2023-12-16T07:41:39.847992+0000 mon.a (mon.0) 877 : audit [DBG] from='client.? 172.21.15.7:0/3480080907' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:41:41.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:41 smithi169 bash[25083]: audit 2023-12-16T07:41:40.207377+0000 mgr.y (mgr.24367) 255 : audit [DBG] from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:41.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:41 smithi169 bash[25083]: cluster 2023-12-16T07:41:40.353800+0000 mgr.y (mgr.24367) 256 : cluster [DBG] pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:41.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:41 smithi169 bash[25083]: audit 2023-12-16T07:41:40.619220+0000 mon.c (mon.1) 34 : audit [DBG] from='client.? 172.21.15.7:0/2717654853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:41:41.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:41 smithi007 bash[25009]: audit 2023-12-16T07:41:40.207377+0000 mgr.y (mgr.24367) 255 : audit [DBG] from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:41.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:41 smithi007 bash[25009]: cluster 2023-12-16T07:41:40.353800+0000 mgr.y (mgr.24367) 256 : cluster [DBG] pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:41.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:41 smithi007 bash[25009]: audit 2023-12-16T07:41:40.619220+0000 mon.c (mon.1) 34 : audit [DBG] from='client.? 172.21.15.7:0/2717654853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:41:41.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:41 smithi007 bash[30436]: audit 2023-12-16T07:41:40.207377+0000 mgr.y (mgr.24367) 255 : audit [DBG] from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:41:41.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:41 smithi007 bash[30436]: cluster 2023-12-16T07:41:40.353800+0000 mgr.y (mgr.24367) 256 : cluster [DBG] pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:41.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:41 smithi007 bash[30436]: audit 2023-12-16T07:41:40.619220+0000 mon.c (mon.1) 34 : audit [DBG] from='client.? 172.21.15.7:0/2717654853' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:41:42.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:41:42 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:41:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:41:43.517 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:43 smithi007 bash[30436]: cluster 2023-12-16T07:41:42.355673+0000 mgr.y (mgr.24367) 257 : cluster [DBG] pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:43.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:43 smithi007 bash[25009]: cluster 2023-12-16T07:41:42.355673+0000 mgr.y (mgr.24367) 257 : cluster [DBG] pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:43.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:43 smithi169 bash[25083]: cluster 2023-12-16T07:41:42.355673+0000 mgr.y (mgr.24367) 257 : cluster [DBG] pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:43.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:41:43 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:41:43] "GET /metrics HTTP/1.1" 200 214841 "" "Prometheus/2.33.4" 2023-12-16T07:41:43.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:41:43 smithi007 bash[49264]: level=warn ts=2023-12-16T07:41:43.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:41:43.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:41:43 smithi007 bash[49264]: level=warn ts=2023-12-16T07:41:43.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2023-12-16T07:41:44.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:44 smithi007 bash[25009]: audit 2023-12-16T07:41:43.727177+0000 mgr.y (mgr.24367) 258 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:44.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:44 smithi007 bash[30436]: audit 2023-12-16T07:41:43.727177+0000 mgr.y (mgr.24367) 258 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:44.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:44 smithi169 bash[25083]: audit 2023-12-16T07:41:43.727177+0000 mgr.y (mgr.24367) 258 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:45.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:45 smithi007 bash[25009]: cluster 2023-12-16T07:41:44.356816+0000 mgr.y (mgr.24367) 259 : cluster [DBG] pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:45.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:45 smithi007 bash[30436]: cluster 2023-12-16T07:41:44.356816+0000 mgr.y (mgr.24367) 259 : cluster [DBG] pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:45.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:45 smithi169 bash[25083]: cluster 2023-12-16T07:41:44.356816+0000 mgr.y (mgr.24367) 259 : cluster [DBG] pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:47.678 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:47 smithi007 bash[30436]: cluster 2023-12-16T07:41:46.358419+0000 mgr.y (mgr.24367) 260 : cluster [DBG] pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:47.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:47 smithi007 bash[25009]: cluster 2023-12-16T07:41:46.358419+0000 mgr.y (mgr.24367) 260 : cluster [DBG] pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:47.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:47 smithi169 bash[25083]: cluster 2023-12-16T07:41:46.358419+0000 mgr.y (mgr.24367) 260 : cluster [DBG] pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[30436]: cluster 2023-12-16T07:41:48.359031+0000 mgr.y (mgr.24367) 261 : cluster [DBG] pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[30436]: audit 2023-12-16T07:41:48.476690+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:41:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[30436]: cephadm 2023-12-16T07:41:48.477382+0000 mgr.y (mgr.24367) 262 : cephadm [INF] Upgrade: Target is version 19.0.0-191-g6d933094 (unknown) 2023-12-16T07:41:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[30436]: cephadm 2023-12-16T07:41:48.477491+0000 mgr.y (mgr.24367) 263 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678, digests ['quay.ceph.io/ceph-ci/ceph@sha256:2829c22c1fdb60b99590a3eef8aa48bb7d6d606e3a4c1627a68d2485fa617e17'] 2023-12-16T07:41:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[30436]: audit 2023-12-16T07:41:48.479125+0000 mon.a (mon.0) 879 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:41:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[30436]: audit 2023-12-16T07:41:48.488618+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:41:49.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[30436]: cephadm 2023-12-16T07:41:48.489825+0000 mgr.y (mgr.24367) 264 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-12-16T07:41:49.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[30436]: cephadm 2023-12-16T07:41:48.733102+0000 mgr.y (mgr.24367) 265 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678 on smithi169 2023-12-16T07:41:49.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[25009]: cluster 2023-12-16T07:41:48.359031+0000 mgr.y (mgr.24367) 261 : cluster [DBG] pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:49.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[25009]: audit 2023-12-16T07:41:48.476690+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:41:49.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[25009]: cephadm 2023-12-16T07:41:48.477382+0000 mgr.y (mgr.24367) 262 : cephadm [INF] Upgrade: Target is version 19.0.0-191-g6d933094 (unknown) 2023-12-16T07:41:49.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[25009]: cephadm 2023-12-16T07:41:48.477491+0000 mgr.y (mgr.24367) 263 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678, digests ['quay.ceph.io/ceph-ci/ceph@sha256:2829c22c1fdb60b99590a3eef8aa48bb7d6d606e3a4c1627a68d2485fa617e17'] 2023-12-16T07:41:49.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[25009]: audit 2023-12-16T07:41:48.479125+0000 mon.a (mon.0) 879 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:41:49.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[25009]: audit 2023-12-16T07:41:48.488618+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:41:49.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[25009]: cephadm 2023-12-16T07:41:48.489825+0000 mgr.y (mgr.24367) 264 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-12-16T07:41:49.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:49 smithi007 bash[25009]: cephadm 2023-12-16T07:41:48.733102+0000 mgr.y (mgr.24367) 265 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678 on smithi169 2023-12-16T07:41:49.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:49 smithi169 bash[25083]: cluster 2023-12-16T07:41:48.359031+0000 mgr.y (mgr.24367) 261 : cluster [DBG] pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:49.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:49 smithi169 bash[25083]: audit 2023-12-16T07:41:48.476690+0000 mon.a (mon.0) 878 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:41:49.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:49 smithi169 bash[25083]: cephadm 2023-12-16T07:41:48.477382+0000 mgr.y (mgr.24367) 262 : cephadm [INF] Upgrade: Target is version 19.0.0-191-g6d933094 (unknown) 2023-12-16T07:41:49.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:49 smithi169 bash[25083]: cephadm 2023-12-16T07:41:48.477491+0000 mgr.y (mgr.24367) 263 : cephadm [INF] Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678, digests ['quay.ceph.io/ceph-ci/ceph@sha256:2829c22c1fdb60b99590a3eef8aa48bb7d6d606e3a4c1627a68d2485fa617e17'] 2023-12-16T07:41:49.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:49 smithi169 bash[25083]: audit 2023-12-16T07:41:48.479125+0000 mon.a (mon.0) 879 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:41:49.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:49 smithi169 bash[25083]: audit 2023-12-16T07:41:48.488618+0000 mon.a (mon.0) 880 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:41:49.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:49 smithi169 bash[25083]: cephadm 2023-12-16T07:41:48.489825+0000 mgr.y (mgr.24367) 264 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-12-16T07:41:49.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:49 smithi169 bash[25083]: cephadm 2023-12-16T07:41:48.733102+0000 mgr.y (mgr.24367) 265 : cephadm [INF] Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678 on smithi169 2023-12-16T07:41:51.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:51 smithi007 bash[25009]: cluster 2023-12-16T07:41:50.359601+0000 mgr.y (mgr.24367) 266 : cluster [DBG] pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:51.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:51 smithi007 bash[25009]: audit 2023-12-16T07:41:50.546281+0000 mon.a (mon.0) 881 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:41:51.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:51 smithi007 bash[25009]: audit 2023-12-16T07:41:50.558431+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:41:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:51 smithi007 bash[30436]: cluster 2023-12-16T07:41:50.359601+0000 mgr.y (mgr.24367) 266 : cluster [DBG] pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:51 smithi007 bash[30436]: audit 2023-12-16T07:41:50.546281+0000 mon.a (mon.0) 881 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:41:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:51 smithi007 bash[30436]: audit 2023-12-16T07:41:50.558431+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:41:51.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:51 smithi169 bash[25083]: cluster 2023-12-16T07:41:50.359601+0000 mgr.y (mgr.24367) 266 : cluster [DBG] pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:51.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:51 smithi169 bash[25083]: audit 2023-12-16T07:41:50.546281+0000 mon.a (mon.0) 881 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:41:51.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:51 smithi169 bash[25083]: audit 2023-12-16T07:41:50.558431+0000 mon.a (mon.0) 882 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:41:52.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:41:52 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:41:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:41:53.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:53 smithi007 bash[25009]: cluster 2023-12-16T07:41:52.361328+0000 mgr.y (mgr.24367) 267 : cluster [DBG] pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:53.514 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:53 smithi007 bash[30436]: cluster 2023-12-16T07:41:52.361328+0000 mgr.y (mgr.24367) 267 : cluster [DBG] pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:53.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:53 smithi169 bash[25083]: cluster 2023-12-16T07:41:52.361328+0000 mgr.y (mgr.24367) 267 : cluster [DBG] pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:53.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:41:53 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:41:53] "GET /metrics HTTP/1.1" 200 214850 "" "Prometheus/2.33.4" 2023-12-16T07:41:53.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:41:53 smithi007 bash[49264]: level=error ts=2023-12-16T07:41:53.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:41:53.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:41:53 smithi007 bash[49264]: level=warn ts=2023-12-16T07:41:53.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:41:53.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:41:53 smithi007 bash[49264]: level=warn ts=2023-12-16T07:41:53.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2023-12-16T07:41:54.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:54 smithi007 bash[25009]: audit 2023-12-16T07:41:53.735122+0000 mgr.y (mgr.24367) 268 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:54.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:54 smithi007 bash[30436]: audit 2023-12-16T07:41:53.735122+0000 mgr.y (mgr.24367) 268 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:54.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:54 smithi169 bash[25083]: audit 2023-12-16T07:41:53.735122+0000 mgr.y (mgr.24367) 268 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:41:55.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:55 smithi007 bash[25009]: cluster 2023-12-16T07:41:54.362082+0000 mgr.y (mgr.24367) 269 : cluster [DBG] pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:55.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:55 smithi007 bash[30436]: cluster 2023-12-16T07:41:54.362082+0000 mgr.y (mgr.24367) 269 : cluster [DBG] pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:55 smithi169 bash[25083]: cluster 2023-12-16T07:41:54.362082+0000 mgr.y (mgr.24367) 269 : cluster [DBG] pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:57.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:57 smithi169 bash[25083]: cluster 2023-12-16T07:41:56.363192+0000 mgr.y (mgr.24367) 270 : cluster [DBG] pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:57.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:57 smithi007 bash[25009]: cluster 2023-12-16T07:41:56.363192+0000 mgr.y (mgr.24367) 270 : cluster [DBG] pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:57.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:57 smithi007 bash[30436]: cluster 2023-12-16T07:41:56.363192+0000 mgr.y (mgr.24367) 270 : cluster [DBG] pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:41:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:41:59 smithi169 bash[25083]: cluster 2023-12-16T07:41:58.363865+0000 mgr.y (mgr.24367) 271 : cluster [DBG] pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:59.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:41:59 smithi007 bash[25009]: cluster 2023-12-16T07:41:58.363865+0000 mgr.y (mgr.24367) 271 : cluster [DBG] pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:41:59.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:41:59 smithi007 bash[30436]: cluster 2023-12-16T07:41:58.363865+0000 mgr.y (mgr.24367) 271 : cluster [DBG] pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:01.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:01 smithi169 bash[25083]: cluster 2023-12-16T07:42:00.364763+0000 mgr.y (mgr.24367) 272 : cluster [DBG] pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:01.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:01 smithi007 bash[25009]: cluster 2023-12-16T07:42:00.364763+0000 mgr.y (mgr.24367) 272 : cluster [DBG] pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:01.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:01 smithi007 bash[30436]: cluster 2023-12-16T07:42:00.364763+0000 mgr.y (mgr.24367) 272 : cluster [DBG] pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:02.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:02 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:42:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:42:03.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:03 smithi007 bash[25009]: cluster 2023-12-16T07:42:02.366129+0000 mgr.y (mgr.24367) 273 : cluster [DBG] pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:03.515 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:03 smithi007 bash[30436]: cluster 2023-12-16T07:42:02.366129+0000 mgr.y (mgr.24367) 273 : cluster [DBG] pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:03.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:03 smithi169 bash[25083]: cluster 2023-12-16T07:42:02.366129+0000 mgr.y (mgr.24367) 273 : cluster [DBG] pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:03.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:03 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:42:03] "GET /metrics HTTP/1.1" 200 214850 "" "Prometheus/2.33.4" 2023-12-16T07:42:03.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:03 smithi007 bash[49264]: level=error ts=2023-12-16T07:42:03.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:42:03.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:03 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:03.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2023-12-16T07:42:03.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:03 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:03.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:42:04.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:04 smithi007 bash[25009]: audit 2023-12-16T07:42:03.743145+0000 mgr.y (mgr.24367) 274 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:04.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:04 smithi007 bash[30436]: audit 2023-12-16T07:42:03.743145+0000 mgr.y (mgr.24367) 274 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:04.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:04 smithi169 bash[25083]: audit 2023-12-16T07:42:03.743145+0000 mgr.y (mgr.24367) 274 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:05.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:05 smithi007 bash[25009]: cluster 2023-12-16T07:42:04.366715+0000 mgr.y (mgr.24367) 275 : cluster [DBG] pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:05.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:05 smithi007 bash[30436]: cluster 2023-12-16T07:42:04.366715+0000 mgr.y (mgr.24367) 275 : cluster [DBG] pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:05.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:05 smithi169 bash[25083]: cluster 2023-12-16T07:42:04.366715+0000 mgr.y (mgr.24367) 275 : cluster [DBG] pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:07.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:07 smithi169 bash[25083]: cluster 2023-12-16T07:42:06.368358+0000 mgr.y (mgr.24367) 276 : cluster [DBG] pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:07.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:07 smithi007 bash[25009]: cluster 2023-12-16T07:42:06.368358+0000 mgr.y (mgr.24367) 276 : cluster [DBG] pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:07.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:07 smithi007 bash[30436]: cluster 2023-12-16T07:42:06.368358+0000 mgr.y (mgr.24367) 276 : cluster [DBG] pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:09.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:09 smithi169 bash[25083]: cluster 2023-12-16T07:42:08.369095+0000 mgr.y (mgr.24367) 277 : cluster [DBG] pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:09.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:09 smithi007 bash[25009]: cluster 2023-12-16T07:42:08.369095+0000 mgr.y (mgr.24367) 277 : cluster [DBG] pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:09.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:09 smithi007 bash[30436]: cluster 2023-12-16T07:42:08.369095+0000 mgr.y (mgr.24367) 277 : cluster [DBG] pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:11.006 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:42:11.724 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:42:11.724 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (5m) 3m ago 5m 16.5M - ba2b418f427c 128c156c0190 2023-12-16T07:42:11.724 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (5m) 3m ago 5m 46.0M - 8.3.5 dad864ee21e9 25b9cf47977e 2023-12-16T07:42:11.724 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (3m) 3m ago 3m 66.0M - 3.5 e1d6a67b021e ca15814d070f 2023-12-16T07:42:11.724 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443 running (11m) 3m ago 11m 405M - 17.2.0 e1d6a67b021e ff78f5dd3700 2023-12-16T07:42:11.724 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:9283 running (13m) 3m ago 13m 452M - 17.2.0 e1d6a67b021e 5219d4698403 2023-12-16T07:42:11.725 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (13m) 3m ago 14m 54.3M 2048M 17.2.0 e1d6a67b021e c0468ecbd44e 2023-12-16T07:42:11.725 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (12m) 3m ago 12m 40.0M 2048M 17.2.0 e1d6a67b021e b47a960c6102 2023-12-16T07:42:11.725 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (12m) 3m ago 12m 42.3M 2048M 17.2.0 e1d6a67b021e c659b986f201 2023-12-16T07:42:11.725 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (5m) 3m ago 6m 11.6M - 1dbe0e931976 54a71535c2b4 2023-12-16T07:42:11.725 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (5m) 3m ago 6m 13.0M - 1dbe0e931976 b050faea525d 2023-12-16T07:42:11.725 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (11m) 3m ago 11m 54.1M 4062M 17.2.0 e1d6a67b021e dbc4d9538238 2023-12-16T07:42:11.725 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (10m) 3m ago 10m 54.6M 4062M 17.2.0 e1d6a67b021e 14637e0f389c 2023-12-16T07:42:11.725 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (10m) 3m ago 10m 53.2M 4062M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:42:11.725 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (9m) 3m ago 9m 50.9M 4062M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:42:11.725 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (8m) 3m ago 8m 51.2M 4317M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:42:11.725 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (8m) 3m ago 8m 50.0M 4317M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:42:11.725 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (7m) 3m ago 7m 47.9M 4317M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:42:11.725 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (6m) 3m ago 6m 50.0M 4317M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:42:11.725 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (4m) 3m ago 6m 49.4M - 514e6a882f6e 9351c4c7d0ba 2023-12-16T07:42:11.726 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (3m) 3m ago 3m 83.5M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:42:11.726 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (3m) 3m ago 3m 83.1M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:42:11.726 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (3m) 3m ago 3m 83.5M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:42:11.726 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (3m) 3m ago 3m 82.8M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:42:11.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:11 smithi007 bash[30436]: cluster 2023-12-16T07:42:10.369809+0000 mgr.y (mgr.24367) 278 : cluster [DBG] pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:11.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:11 smithi007 bash[30436]: audit 2023-12-16T07:42:10.988619+0000 mgr.y (mgr.24367) 279 : audit [DBG] from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:11 smithi007 bash[25009]: cluster 2023-12-16T07:42:10.369809+0000 mgr.y (mgr.24367) 278 : cluster [DBG] pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:11 smithi007 bash[25009]: audit 2023-12-16T07:42:10.988619+0000 mgr.y (mgr.24367) 279 : audit [DBG] from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:11 smithi169 bash[25083]: cluster 2023-12-16T07:42:10.369809+0000 mgr.y (mgr.24367) 278 : cluster [DBG] pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:11 smithi169 bash[25083]: audit 2023-12-16T07:42:10.988619+0000 mgr.y (mgr.24367) 279 : audit [DBG] from='client.25057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:12.141 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:42:12.141 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:42:12.141 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2023-12-16T07:42:12.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:42:12.141 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:42:12.141 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2023-12-16T07:42:12.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:42:12.141 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:42:12.141 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2023-12-16T07:42:12.141 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:42:12.141 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2023-12-16T07:42:12.141 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:42:12.142 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:42:12.142 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:42:12.142 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:42:12.142 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2023-12-16T07:42:12.142 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:42:12.142 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:42:12.438 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:12 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:42:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:42:12.502 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:42:12.502 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:42:12.502 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:42:12.502 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [], 2023-12-16T07:42:12.502 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "0/23 daemons upgraded", 2023-12-16T07:42:12.502 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678 image on host smithi169" 2023-12-16T07:42:12.502 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:42:12.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:12 smithi007 bash[30436]: audit 2023-12-16T07:42:11.353232+0000 mgr.y (mgr.24367) 280 : audit [DBG] from='client.24938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:12.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:12 smithi007 bash[30436]: audit 2023-12-16T07:42:11.699696+0000 mgr.y (mgr.24367) 281 : audit [DBG] from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:12.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:12 smithi007 bash[30436]: audit 2023-12-16T07:42:12.138822+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.7:0/3299182349' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:42:12.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:12 smithi007 bash[25009]: audit 2023-12-16T07:42:11.353232+0000 mgr.y (mgr.24367) 280 : audit [DBG] from='client.24938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:12.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:12 smithi007 bash[25009]: audit 2023-12-16T07:42:11.699696+0000 mgr.y (mgr.24367) 281 : audit [DBG] from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:12.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:12 smithi007 bash[25009]: audit 2023-12-16T07:42:12.138822+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.7:0/3299182349' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:42:12.916 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:42:12.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:12 smithi169 bash[25083]: audit 2023-12-16T07:42:11.353232+0000 mgr.y (mgr.24367) 280 : audit [DBG] from='client.24938 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:12.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:12 smithi169 bash[25083]: audit 2023-12-16T07:42:11.699696+0000 mgr.y (mgr.24367) 281 : audit [DBG] from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:12.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:12 smithi169 bash[25083]: audit 2023-12-16T07:42:12.138822+0000 mon.c (mon.1) 35 : audit [DBG] from='client.? 172.21.15.7:0/3299182349' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:42:13.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:13 smithi007 bash[30436]: cluster 2023-12-16T07:42:12.371470+0000 mgr.y (mgr.24367) 282 : cluster [DBG] pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:13.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:13 smithi007 bash[30436]: audit 2023-12-16T07:42:12.499527+0000 mgr.y (mgr.24367) 283 : audit [DBG] from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:13.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:13 smithi007 bash[30436]: audit 2023-12-16T07:42:12.914327+0000 mon.a (mon.0) 883 : audit [DBG] from='client.? 172.21.15.7:0/913339785' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:42:13.717 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:13 smithi007 bash[49264]: level=error ts=2023-12-16T07:42:13.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:42:13.717 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:13 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:13.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:42:13.718 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:13 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:13.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2023-12-16T07:42:13.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:13 smithi007 bash[25009]: cluster 2023-12-16T07:42:12.371470+0000 mgr.y (mgr.24367) 282 : cluster [DBG] pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:13.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:13 smithi007 bash[25009]: audit 2023-12-16T07:42:12.499527+0000 mgr.y (mgr.24367) 283 : audit [DBG] from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:13.718 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:13 smithi007 bash[25009]: audit 2023-12-16T07:42:12.914327+0000 mon.a (mon.0) 883 : audit [DBG] from='client.? 172.21.15.7:0/913339785' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:42:13.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:13 smithi169 bash[25083]: cluster 2023-12-16T07:42:12.371470+0000 mgr.y (mgr.24367) 282 : cluster [DBG] pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:13.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:13 smithi169 bash[25083]: audit 2023-12-16T07:42:12.499527+0000 mgr.y (mgr.24367) 283 : audit [DBG] from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:13.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:13 smithi169 bash[25083]: audit 2023-12-16T07:42:12.914327+0000 mon.a (mon.0) 883 : audit [DBG] from='client.? 172.21.15.7:0/913339785' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:42:14.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:13 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:42:13] "GET /metrics HTTP/1.1" 200 214837 "" "Prometheus/2.33.4" 2023-12-16T07:42:14.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:14 smithi007 bash[25009]: audit 2023-12-16T07:42:13.751131+0000 mgr.y (mgr.24367) 284 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:14.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:14 smithi007 bash[30436]: audit 2023-12-16T07:42:13.751131+0000 mgr.y (mgr.24367) 284 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:14.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:14 smithi169 bash[25083]: audit 2023-12-16T07:42:13.751131+0000 mgr.y (mgr.24367) 284 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:15.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:15 smithi007 bash[25009]: cluster 2023-12-16T07:42:14.372339+0000 mgr.y (mgr.24367) 285 : cluster [DBG] pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:15.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:15 smithi007 bash[30436]: cluster 2023-12-16T07:42:14.372339+0000 mgr.y (mgr.24367) 285 : cluster [DBG] pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:15.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:15 smithi169 bash[25083]: cluster 2023-12-16T07:42:14.372339+0000 mgr.y (mgr.24367) 285 : cluster [DBG] pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:17.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:17 smithi007 bash[25009]: cluster 2023-12-16T07:42:16.374061+0000 mgr.y (mgr.24367) 286 : cluster [DBG] pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:17.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:17 smithi007 bash[30436]: cluster 2023-12-16T07:42:16.374061+0000 mgr.y (mgr.24367) 286 : cluster [DBG] pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:17.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:17 smithi169 bash[25083]: cluster 2023-12-16T07:42:16.374061+0000 mgr.y (mgr.24367) 286 : cluster [DBG] pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:19.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:19 smithi007 bash[25009]: cluster 2023-12-16T07:42:18.374822+0000 mgr.y (mgr.24367) 287 : cluster [DBG] pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:19.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:19 smithi007 bash[30436]: cluster 2023-12-16T07:42:18.374822+0000 mgr.y (mgr.24367) 287 : cluster [DBG] pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:19.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:19 smithi169 bash[25083]: cluster 2023-12-16T07:42:18.374822+0000 mgr.y (mgr.24367) 287 : cluster [DBG] pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:21.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:21 smithi007 bash[25009]: cluster 2023-12-16T07:42:20.375595+0000 mgr.y (mgr.24367) 288 : cluster [DBG] pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:21.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:21 smithi007 bash[30436]: cluster 2023-12-16T07:42:20.375595+0000 mgr.y (mgr.24367) 288 : cluster [DBG] pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:21.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:21 smithi169 bash[25083]: cluster 2023-12-16T07:42:20.375595+0000 mgr.y (mgr.24367) 288 : cluster [DBG] pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:22.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:22 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:42:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:42:23.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:23 smithi007 bash[30436]: cluster 2023-12-16T07:42:22.377689+0000 mgr.y (mgr.24367) 289 : cluster [DBG] pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:23.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:23 smithi007 bash[25009]: cluster 2023-12-16T07:42:22.377689+0000 mgr.y (mgr.24367) 289 : cluster [DBG] pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:23.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:23 smithi169 bash[25083]: cluster 2023-12-16T07:42:22.377689+0000 mgr.y (mgr.24367) 289 : cluster [DBG] pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:23.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:23 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:42:23] "GET /metrics HTTP/1.1" 200 214848 "" "Prometheus/2.33.4" 2023-12-16T07:42:23.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:23 smithi007 bash[49264]: level=error ts=2023-12-16T07:42:23.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2023-12-16T07:42:23.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:23 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:23.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:42:23.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:23 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:23.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2023-12-16T07:42:24.556 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:24 smithi169 bash[25083]: audit 2023-12-16T07:42:23.762364+0000 mgr.y (mgr.24367) 290 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:24.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:24 smithi007 bash[25009]: audit 2023-12-16T07:42:23.762364+0000 mgr.y (mgr.24367) 290 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:24.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:24 smithi007 bash[30436]: audit 2023-12-16T07:42:23.762364+0000 mgr.y (mgr.24367) 290 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:25.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:25 smithi007 bash[25009]: cluster 2023-12-16T07:42:24.378444+0000 mgr.y (mgr.24367) 291 : cluster [DBG] pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:25.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:25 smithi007 bash[30436]: cluster 2023-12-16T07:42:24.378444+0000 mgr.y (mgr.24367) 291 : cluster [DBG] pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:25 smithi169 bash[25083]: cluster 2023-12-16T07:42:24.378444+0000 mgr.y (mgr.24367) 291 : cluster [DBG] pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:27.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:27 smithi007 bash[25009]: cluster 2023-12-16T07:42:26.379827+0000 mgr.y (mgr.24367) 292 : cluster [DBG] pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:27.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:27 smithi007 bash[30436]: cluster 2023-12-16T07:42:26.379827+0000 mgr.y (mgr.24367) 292 : cluster [DBG] pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:27.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:27 smithi169 bash[25083]: cluster 2023-12-16T07:42:26.379827+0000 mgr.y (mgr.24367) 292 : cluster [DBG] pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:29.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:29 smithi007 bash[25009]: cluster 2023-12-16T07:42:28.380698+0000 mgr.y (mgr.24367) 293 : cluster [DBG] pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:29.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:29 smithi007 bash[30436]: cluster 2023-12-16T07:42:28.380698+0000 mgr.y (mgr.24367) 293 : cluster [DBG] pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:29.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:29 smithi169 bash[25083]: cluster 2023-12-16T07:42:28.380698+0000 mgr.y (mgr.24367) 293 : cluster [DBG] pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:31.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:31 smithi007 bash[25009]: cluster 2023-12-16T07:42:30.381596+0000 mgr.y (mgr.24367) 294 : cluster [DBG] pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:31.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:31 smithi007 bash[30436]: cluster 2023-12-16T07:42:30.381596+0000 mgr.y (mgr.24367) 294 : cluster [DBG] pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:31.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:31 smithi169 bash[25083]: cluster 2023-12-16T07:42:30.381596+0000 mgr.y (mgr.24367) 294 : cluster [DBG] pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:32.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:32 smithi169 bash[25855]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:42:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:42:33.517 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:33 smithi007 bash[30436]: cluster 2023-12-16T07:42:32.383375+0000 mgr.y (mgr.24367) 295 : cluster [DBG] pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:33.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:33 smithi007 bash[25009]: cluster 2023-12-16T07:42:32.383375+0000 mgr.y (mgr.24367) 295 : cluster [DBG] pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:33.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:33 smithi169 bash[25083]: cluster 2023-12-16T07:42:32.383375+0000 mgr.y (mgr.24367) 295 : cluster [DBG] pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:33.773 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:33 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:42:33] "GET /metrics HTTP/1.1" 200 214848 "" "Prometheus/2.33.4" 2023-12-16T07:42:33.773 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:33 smithi007 bash[49264]: level=error ts=2023-12-16T07:42:33.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:42:33.774 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:33 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:33.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2023-12-16T07:42:33.774 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:33 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:33.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:42:34.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:34 smithi007 bash[30436]: audit 2023-12-16T07:42:33.771080+0000 mgr.y (mgr.24367) 296 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:34.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:34 smithi007 bash[25009]: audit 2023-12-16T07:42:33.771080+0000 mgr.y (mgr.24367) 296 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:34 smithi169 bash[25083]: audit 2023-12-16T07:42:33.771080+0000 mgr.y (mgr.24367) 296 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:35.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:35 smithi007 bash[25009]: cluster 2023-12-16T07:42:34.384106+0000 mgr.y (mgr.24367) 297 : cluster [DBG] pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:35.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:35 smithi007 bash[30436]: cluster 2023-12-16T07:42:34.384106+0000 mgr.y (mgr.24367) 297 : cluster [DBG] pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:35.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:35 smithi169 bash[25083]: cluster 2023-12-16T07:42:34.384106+0000 mgr.y (mgr.24367) 297 : cluster [DBG] pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:37.546 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:37 smithi169 bash[25083]: cluster 2023-12-16T07:42:36.385730+0000 mgr.y (mgr.24367) 298 : cluster [DBG] pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:37.547 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:37 smithi169 bash[25083]: audit 2023-12-16T07:42:37.308878+0000 mon.a (mon.0) 884 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:37.547 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:37 smithi169 bash[25083]: audit 2023-12-16T07:42:37.310753+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:42:37.547 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:37 smithi169 bash[25083]: audit 2023-12-16T07:42:37.312768+0000 mon.a (mon.0) 886 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:42:37.547 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:37 smithi169 bash[25083]: audit 2023-12-16T07:42:37.314877+0000 mon.a (mon.0) 887 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:42:37.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:37 smithi007 bash[25009]: cluster 2023-12-16T07:42:36.385730+0000 mgr.y (mgr.24367) 298 : cluster [DBG] pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:37.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:37 smithi007 bash[25009]: audit 2023-12-16T07:42:37.308878+0000 mon.a (mon.0) 884 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:37.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:37 smithi007 bash[25009]: audit 2023-12-16T07:42:37.310753+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:42:37.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:37 smithi007 bash[25009]: audit 2023-12-16T07:42:37.312768+0000 mon.a (mon.0) 886 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:42:37.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:37 smithi007 bash[25009]: audit 2023-12-16T07:42:37.314877+0000 mon.a (mon.0) 887 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:42:37.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:37 smithi007 bash[30436]: cluster 2023-12-16T07:42:36.385730+0000 mgr.y (mgr.24367) 298 : cluster [DBG] pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:37.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:37 smithi007 bash[30436]: audit 2023-12-16T07:42:37.308878+0000 mon.a (mon.0) 884 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:37.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:37 smithi007 bash[30436]: audit 2023-12-16T07:42:37.310753+0000 mon.a (mon.0) 885 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:42:37.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:37 smithi007 bash[30436]: audit 2023-12-16T07:42:37.312768+0000 mon.a (mon.0) 886 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:42:37.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:37 smithi007 bash[30436]: audit 2023-12-16T07:42:37.314877+0000 mon.a (mon.0) 887 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:42:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:38 smithi169 bash[25083]: cephadm 2023-12-16T07:42:37.299422+0000 mgr.y (mgr.24367) 299 : cephadm [INF] Upgrade: Updating mgr.x 2023-12-16T07:42:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:38 smithi169 bash[25083]: cephadm 2023-12-16T07:42:37.316518+0000 mgr.y (mgr.24367) 300 : cephadm [INF] Deploying daemon mgr.x on smithi169 2023-12-16T07:42:38.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:38 smithi007 bash[25009]: cephadm 2023-12-16T07:42:37.299422+0000 mgr.y (mgr.24367) 299 : cephadm [INF] Upgrade: Updating mgr.x 2023-12-16T07:42:38.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:38 smithi007 bash[25009]: cephadm 2023-12-16T07:42:37.316518+0000 mgr.y (mgr.24367) 300 : cephadm [INF] Deploying daemon mgr.x on smithi169 2023-12-16T07:42:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:38 smithi007 bash[30436]: cephadm 2023-12-16T07:42:37.299422+0000 mgr.y (mgr.24367) 299 : cephadm [INF] Upgrade: Updating mgr.x 2023-12-16T07:42:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:38 smithi007 bash[30436]: cephadm 2023-12-16T07:42:37.316518+0000 mgr.y (mgr.24367) 300 : cephadm [INF] Deploying daemon mgr.x on smithi169 2023-12-16T07:42:39.668 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:39 smithi169 bash[25083]: cluster 2023-12-16T07:42:38.386419+0000 mgr.y (mgr.24367) 301 : cluster [DBG] pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:39.668 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:39 smithi169 systemd[1]: Stopping Ceph mgr.x for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:42:39.668 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:39 smithi169 bash[45062]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mgr.x 2023-12-16T07:42:39.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:39 smithi007 bash[25009]: cluster 2023-12-16T07:42:38.386419+0000 mgr.y (mgr.24367) 301 : cluster [DBG] pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:39.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:39 smithi007 bash[30436]: cluster 2023-12-16T07:42:38.386419+0000 mgr.y (mgr.24367) 301 : cluster [DBG] pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:40.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:40 smithi169 bash[45072]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mgr-x 2023-12-16T07:42:40.434 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:40 smithi169 bash[45119]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mgr.x 2023-12-16T07:42:40.434 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:40 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2023-12-16T07:42:40.434 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:40 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2023-12-16T07:42:40.434 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:40 smithi169 systemd[1]: Stopped Ceph mgr.x for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:42:40.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:40 smithi169 systemd[1]: Started Ceph mgr.x for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:42:41.860 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:41 smithi169 bash[45188]: debug 2023-12-16T07:42:41.697+0000 7fd331832700 1 -- 172.21.15.169:0/2899269348 <== mon.2 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5578d9267520 con 0x5578d9fe4400 2023-12-16T07:42:41.860 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:41 smithi169 bash[25083]: cluster 2023-12-16T07:42:40.387156+0000 mgr.y (mgr.24367) 302 : cluster [DBG] pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:41.860 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:41 smithi169 bash[25083]: audit 2023-12-16T07:42:40.634759+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:41.860 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:41 smithi169 bash[25083]: audit 2023-12-16T07:42:40.636787+0000 mon.a (mon.0) 889 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:42:41.860 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:41 smithi169 bash[25083]: audit 2023-12-16T07:42:40.638201+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:42:42.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:41 smithi007 bash[25009]: cluster 2023-12-16T07:42:40.387156+0000 mgr.y (mgr.24367) 302 : cluster [DBG] pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:42.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:41 smithi007 bash[25009]: audit 2023-12-16T07:42:40.634759+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:42.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:41 smithi007 bash[25009]: audit 2023-12-16T07:42:40.636787+0000 mon.a (mon.0) 889 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:42:42.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:41 smithi007 bash[25009]: audit 2023-12-16T07:42:40.638201+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:42:42.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:41 smithi007 bash[30436]: cluster 2023-12-16T07:42:40.387156+0000 mgr.y (mgr.24367) 302 : cluster [DBG] pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:42.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:41 smithi007 bash[30436]: audit 2023-12-16T07:42:40.634759+0000 mon.a (mon.0) 888 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:42.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:41 smithi007 bash[30436]: audit 2023-12-16T07:42:40.636787+0000 mon.a (mon.0) 889 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:42:42.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:41 smithi007 bash[30436]: audit 2023-12-16T07:42:40.638201+0000 mon.a (mon.0) 890 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:42:42.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:41 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:41.789Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused" 2023-12-16T07:42:42.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:41 smithi169 bash[45188]: debug 2023-12-16T07:42:41.857+0000 7fd33b9d9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T07:42:43.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:42 smithi169 bash[45188]: debug 2023-12-16T07:42:42.761+0000 7fd33b9d9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-16T07:42:43.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:42 smithi169 bash[45188]: debug 2023-12-16T07:42:42.861+0000 7fd33b9d9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T07:42:43.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:42 smithi169 bash[45188]: debug 2023-12-16T07:42:42.965+0000 7fd33b9d9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T07:42:43.304 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:42:43.518 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:43 smithi007 bash[30436]: cluster 2023-12-16T07:42:42.388898+0000 mgr.y (mgr.24367) 303 : cluster [DBG] pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:43.519 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:43 smithi007 bash[25009]: cluster 2023-12-16T07:42:42.388898+0000 mgr.y (mgr.24367) 303 : cluster [DBG] pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:43.623 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:43 smithi169 bash[25083]: cluster 2023-12-16T07:42:42.388898+0000 mgr.y (mgr.24367) 303 : cluster [DBG] pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:43.624 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:43 smithi169 bash[45188]: debug 2023-12-16T07:42:43.241+0000 7fd33b9d9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:42:43.624 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:43 smithi169 bash[45188]: debug 2023-12-16T07:42:43.621+0000 7fd33b9d9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T07:42:43.781 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:43 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:42:43] "GET /metrics HTTP/1.1" 200 214840 "" "Prometheus/2.33.4" 2023-12-16T07:42:43.781 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:43 smithi007 bash[49264]: level=error ts=2023-12-16T07:42:43.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.169:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused" 2023-12-16T07:42:43.781 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:43 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:43.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused" 2023-12-16T07:42:43.781 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:43 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:43.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2023-12-16T07:42:43.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:43 smithi169 bash[45188]: debug 2023-12-16T07:42:43.789+0000 7fd33b9d9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T07:42:44.018 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:42:44.018 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (5m) 3m ago 6m 16.5M - ba2b418f427c 128c156c0190 2023-12-16T07:42:44.018 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (5m) 4m ago 5m 46.0M - 8.3.5 dad864ee21e9 25b9cf47977e 2023-12-16T07:42:44.018 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (4m) 3m ago 4m 66.0M - 3.5 e1d6a67b021e ca15814d070f 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283 starting - - - - 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:9283 running (14m) 3m ago 14m 452M - 17.2.0 e1d6a67b021e 5219d4698403 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (14m) 3m ago 14m 54.3M 2048M 17.2.0 e1d6a67b021e c0468ecbd44e 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (12m) 4m ago 12m 40.0M 2048M 17.2.0 e1d6a67b021e b47a960c6102 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (12m) 3m ago 12m 42.3M 2048M 17.2.0 e1d6a67b021e c659b986f201 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (6m) 3m ago 6m 11.6M - 1dbe0e931976 54a71535c2b4 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (6m) 4m ago 6m 13.0M - 1dbe0e931976 b050faea525d 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (11m) 3m ago 11m 54.1M 4062M 17.2.0 e1d6a67b021e dbc4d9538238 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (11m) 3m ago 11m 54.6M 4062M 17.2.0 e1d6a67b021e 14637e0f389c 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (10m) 3m ago 10m 53.2M 4062M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (9m) 3m ago 9m 50.9M 4062M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (9m) 4m ago 9m 51.2M 4317M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (8m) 4m ago 8m 50.0M 4317M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (8m) 4m ago 8m 47.9M 4317M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:42:44.019 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (7m) 4m ago 7m 50.0M 4317M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:42:44.020 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (5m) 4m ago 6m 49.4M - 514e6a882f6e 9351c4c7d0ba 2023-12-16T07:42:44.020 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (4m) 3m ago 4m 83.5M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:42:44.020 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (4m) 4m ago 4m 83.1M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:42:44.020 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (4m) 3m ago 4m 83.5M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:42:44.020 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (4m) 4m ago 4m 82.8M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:42:44.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:44 smithi169 bash[45188]: debug 2023-12-16T07:42:44.009+0000 7fd33b9d9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T07:42:44.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:44 smithi169 bash[25083]: audit 2023-12-16T07:42:43.286965+0000 mgr.y (mgr.24367) 304 : audit [DBG] from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:44.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:44 smithi169 bash[25083]: audit 2023-12-16T07:42:43.642470+0000 mgr.y (mgr.24367) 305 : audit [DBG] from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:44.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:44 smithi169 bash[25083]: audit 2023-12-16T07:42:43.778908+0000 mgr.y (mgr.24367) 306 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:44.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:44 smithi169 bash[25083]: audit 2023-12-16T07:42:43.993965+0000 mgr.y (mgr.24367) 307 : audit [DBG] from='client.25111 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:44.437 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:42:44.437 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:42:44.437 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2023-12-16T07:42:44.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:42:44.437 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:42:44.437 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2023-12-16T07:42:44.437 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:42:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:42:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2023-12-16T07:42:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:42:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2023-12-16T07:42:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:42:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:42:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:42:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:42:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2023-12-16T07:42:44.438 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:42:44.438 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:42:44.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:44 smithi007 bash[30436]: audit 2023-12-16T07:42:43.286965+0000 mgr.y (mgr.24367) 304 : audit [DBG] from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:44.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:44 smithi007 bash[30436]: audit 2023-12-16T07:42:43.642470+0000 mgr.y (mgr.24367) 305 : audit [DBG] from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:44.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:44 smithi007 bash[30436]: audit 2023-12-16T07:42:43.778908+0000 mgr.y (mgr.24367) 306 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:44.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:44 smithi007 bash[30436]: audit 2023-12-16T07:42:43.993965+0000 mgr.y (mgr.24367) 307 : audit [DBG] from='client.25111 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:44.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:44 smithi007 bash[25009]: audit 2023-12-16T07:42:43.286965+0000 mgr.y (mgr.24367) 304 : audit [DBG] from='client.15189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:44.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:44 smithi007 bash[25009]: audit 2023-12-16T07:42:43.642470+0000 mgr.y (mgr.24367) 305 : audit [DBG] from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:44.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:44 smithi007 bash[25009]: audit 2023-12-16T07:42:43.778908+0000 mgr.y (mgr.24367) 306 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:42:44.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:44 smithi007 bash[25009]: audit 2023-12-16T07:42:43.993965+0000 mgr.y (mgr.24367) 307 : audit [DBG] from='client.25111 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:44.800 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:42:44.800 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:42:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:42:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [], 2023-12-16T07:42:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "1/23 daemons upgraded", 2023-12-16T07:42:44.801 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading mgr daemons" 2023-12-16T07:42:44.801 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:42:45.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:45 smithi169 bash[45188]: debug 2023-12-16T07:42:45.113+0000 7fd33b9d9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T07:42:45.227 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:42:45.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:45 smithi007 bash[30436]: cluster 2023-12-16T07:42:44.389619+0000 mgr.y (mgr.24367) 308 : cluster [DBG] pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:45.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:45 smithi007 bash[30436]: audit 2023-12-16T07:42:44.435448+0000 mon.c (mon.1) 36 : audit [DBG] from='client.? 172.21.15.7:0/1602290424' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:42:45.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:45 smithi007 bash[30436]: audit 2023-12-16T07:42:44.624076+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:45.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:45 smithi007 bash[30436]: audit 2023-12-16T07:42:44.797919+0000 mgr.y (mgr.24367) 309 : audit [DBG] from='client.25123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:45.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:45 smithi007 bash[30436]: audit 2023-12-16T07:42:45.092180+0000 mon.a (mon.0) 892 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:45.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:45 smithi007 bash[30436]: audit 2023-12-16T07:42:45.225723+0000 mon.a (mon.0) 893 : audit [DBG] from='client.? 172.21.15.7:0/164310648' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:42:45.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:45 smithi007 bash[25009]: cluster 2023-12-16T07:42:44.389619+0000 mgr.y (mgr.24367) 308 : cluster [DBG] pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:45.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:45 smithi007 bash[25009]: audit 2023-12-16T07:42:44.435448+0000 mon.c (mon.1) 36 : audit [DBG] from='client.? 172.21.15.7:0/1602290424' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:42:45.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:45 smithi007 bash[25009]: audit 2023-12-16T07:42:44.624076+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:45.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:45 smithi007 bash[25009]: audit 2023-12-16T07:42:44.797919+0000 mgr.y (mgr.24367) 309 : audit [DBG] from='client.25123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:45.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:45 smithi007 bash[25009]: audit 2023-12-16T07:42:45.092180+0000 mon.a (mon.0) 892 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:45.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:45 smithi007 bash[25009]: audit 2023-12-16T07:42:45.225723+0000 mon.a (mon.0) 893 : audit [DBG] from='client.? 172.21.15.7:0/164310648' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:42:45.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:45 smithi169 bash[25083]: cluster 2023-12-16T07:42:44.389619+0000 mgr.y (mgr.24367) 308 : cluster [DBG] pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:45.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:45 smithi169 bash[25083]: audit 2023-12-16T07:42:44.435448+0000 mon.c (mon.1) 36 : audit [DBG] from='client.? 172.21.15.7:0/1602290424' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:42:45.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:45 smithi169 bash[25083]: audit 2023-12-16T07:42:44.624076+0000 mon.a (mon.0) 891 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:45.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:45 smithi169 bash[25083]: audit 2023-12-16T07:42:44.797919+0000 mgr.y (mgr.24367) 309 : audit [DBG] from='client.25123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:42:45.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:45 smithi169 bash[25083]: audit 2023-12-16T07:42:45.092180+0000 mon.a (mon.0) 892 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:45.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:45 smithi169 bash[25083]: audit 2023-12-16T07:42:45.225723+0000 mon.a (mon.0) 893 : audit [DBG] from='client.? 172.21.15.7:0/164310648' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:42:45.684 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:45 smithi169 bash[45188]: debug 2023-12-16T07:42:45.233+0000 7fd33b9d9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T07:42:45.684 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:45 smithi169 bash[45188]: debug 2023-12-16T07:42:45.353+0000 7fd33b9d9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T07:42:46.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:45 smithi169 bash[45188]: debug 2023-12-16T07:42:45.825+0000 7fd33b9d9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T07:42:46.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:46 smithi169 bash[45188]: debug 2023-12-16T07:42:46.045+0000 7fd33b9d9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T07:42:46.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:46 smithi169 bash[45188]: debug 2023-12-16T07:42:46.501+0000 7fd33b9d9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T07:42:47.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:46 smithi169 bash[45188]: debug 2023-12-16T07:42:46.989+0000 7fd33b9d9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:42:47.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:47 smithi007 bash[25009]: cluster 2023-12-16T07:42:46.390746+0000 mgr.y (mgr.24367) 310 : cluster [DBG] pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:47.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:47 smithi007 bash[30436]: cluster 2023-12-16T07:42:46.390746+0000 mgr.y (mgr.24367) 310 : cluster [DBG] pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:47.876 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:47 smithi169 bash[25083]: cluster 2023-12-16T07:42:46.390746+0000 mgr.y (mgr.24367) 310 : cluster [DBG] pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:47.876 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:47 smithi169 bash[45188]: debug 2023-12-16T07:42:47.533+0000 7fd33b9d9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T07:42:47.876 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:47 smithi169 bash[45188]: debug 2023-12-16T07:42:47.641+0000 7fd33b9d9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T07:42:48.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:47 smithi169 bash[45188]: debug 2023-12-16T07:42:47.985+0000 7fd33b9d9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T07:42:48.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:48 smithi169 bash[45188]: debug 2023-12-16T07:42:48.097+0000 7fd33b9d9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T07:42:48.495 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:48 smithi169 bash[45188]: debug 2023-12-16T07:42:48.241+0000 7fd33b9d9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T07:42:48.495 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:48 smithi169 bash[45188]: debug 2023-12-16T07:42:48.357+0000 7fd33b9d9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T07:42:48.837 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:48 smithi169 bash[45188]: debug 2023-12-16T07:42:48.493+0000 7fd33b9d9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T07:42:48.837 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:48 smithi169 bash[45188]: debug 2023-12-16T07:42:48.697+0000 7fd33b9d9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T07:42:49.147 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:48 smithi169 bash[45188]: debug 2023-12-16T07:42:48.833+0000 7fd33b9d9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T07:42:49.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:49 smithi169 bash[25083]: cluster 2023-12-16T07:42:48.391458+0000 mgr.y (mgr.24367) 311 : cluster [DBG] pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:49.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:49 smithi007 bash[25009]: cluster 2023-12-16T07:42:48.391458+0000 mgr.y (mgr.24367) 311 : cluster [DBG] pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:49 smithi007 bash[30436]: cluster 2023-12-16T07:42:48.391458+0000 mgr.y (mgr.24367) 311 : cluster [DBG] pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:50.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:50 smithi169 bash[45188]: debug 2023-12-16T07:42:50.101+0000 7fd33b9d9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T07:42:50.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:50 smithi169 bash[45188]: [16/Dec/2023:07:42:50] ENGINE Bus STARTING 2023-12-16T07:42:50.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:50 smithi169 bash[45188]: CherryPy Checker: 2023-12-16T07:42:50.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:50 smithi169 bash[45188]: The Application mounted at '' has an empty config. 2023-12-16T07:42:50.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:50 smithi169 bash[45188]: [16/Dec/2023:07:42:50] ENGINE Serving on http://:::9283 2023-12-16T07:42:50.434 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:50 smithi169 bash[45188]: [16/Dec/2023:07:42:50] ENGINE Bus STARTED 2023-12-16T07:42:50.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:50 smithi007 bash[25009]: cluster 2023-12-16T07:42:50.108147+0000 mon.a (mon.0) 894 : cluster [DBG] Standby manager daemon x restarted 2023-12-16T07:42:50.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:50 smithi007 bash[25009]: cluster 2023-12-16T07:42:50.108394+0000 mon.a (mon.0) 895 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:42:50.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:50 smithi007 bash[25009]: audit 2023-12-16T07:42:50.113315+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:42:50.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:50 smithi007 bash[25009]: audit 2023-12-16T07:42:50.115874+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.? 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:42:50.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:50 smithi007 bash[25009]: audit 2023-12-16T07:42:50.121226+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.? 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:42:50.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:50 smithi007 bash[25009]: audit 2023-12-16T07:42:50.122141+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.? 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:42:50.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:50 smithi007 bash[30436]: cluster 2023-12-16T07:42:50.108147+0000 mon.a (mon.0) 894 : cluster [DBG] Standby manager daemon x restarted 2023-12-16T07:42:50.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:50 smithi007 bash[30436]: cluster 2023-12-16T07:42:50.108394+0000 mon.a (mon.0) 895 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:42:50.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:50 smithi007 bash[30436]: audit 2023-12-16T07:42:50.113315+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:42:50.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:50 smithi007 bash[30436]: audit 2023-12-16T07:42:50.115874+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.? 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:42:50.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:50 smithi007 bash[30436]: audit 2023-12-16T07:42:50.121226+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.? 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:42:50.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:50 smithi007 bash[30436]: audit 2023-12-16T07:42:50.122141+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.? 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:42:50.864 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:50 smithi169 bash[25083]: cluster 2023-12-16T07:42:50.108147+0000 mon.a (mon.0) 894 : cluster [DBG] Standby manager daemon x restarted 2023-12-16T07:42:50.864 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:50 smithi169 bash[25083]: cluster 2023-12-16T07:42:50.108394+0000 mon.a (mon.0) 895 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:42:50.864 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:50 smithi169 bash[25083]: audit 2023-12-16T07:42:50.113315+0000 mon.b (mon.2) 25 : audit [DBG] from='mgr.? 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:42:50.864 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:50 smithi169 bash[25083]: audit 2023-12-16T07:42:50.115874+0000 mon.b (mon.2) 26 : audit [DBG] from='mgr.? 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:42:50.864 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:50 smithi169 bash[25083]: audit 2023-12-16T07:42:50.121226+0000 mon.b (mon.2) 27 : audit [DBG] from='mgr.? 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:42:50.864 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:50 smithi169 bash[25083]: audit 2023-12-16T07:42:50.122141+0000 mon.b (mon.2) 28 : audit [DBG] from='mgr.? 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:42:51.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:51 smithi007 bash[25009]: cluster 2023-12-16T07:42:50.392233+0000 mgr.y (mgr.24367) 312 : cluster [DBG] pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:51.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:51 smithi007 bash[25009]: cluster 2023-12-16T07:42:50.476676+0000 mon.a (mon.0) 896 : cluster [DBG] mgrmap e22: y(active, since 7m), standbys: x 2023-12-16T07:42:51.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:51 smithi007 bash[25009]: audit 2023-12-16T07:42:50.550887+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:42:51.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:51 smithi007 bash[25009]: audit 2023-12-16T07:42:50.563934+0000 mon.a (mon.0) 898 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:42:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:51 smithi007 bash[30436]: cluster 2023-12-16T07:42:50.392233+0000 mgr.y (mgr.24367) 312 : cluster [DBG] pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:51 smithi007 bash[30436]: cluster 2023-12-16T07:42:50.476676+0000 mon.a (mon.0) 896 : cluster [DBG] mgrmap e22: y(active, since 7m), standbys: x 2023-12-16T07:42:51.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:51 smithi007 bash[30436]: audit 2023-12-16T07:42:50.550887+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:42:51.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:51 smithi007 bash[30436]: audit 2023-12-16T07:42:50.563934+0000 mon.a (mon.0) 898 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:42:51.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:51 smithi169 bash[25083]: cluster 2023-12-16T07:42:50.392233+0000 mgr.y (mgr.24367) 312 : cluster [DBG] pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:42:51.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:51 smithi169 bash[25083]: cluster 2023-12-16T07:42:50.476676+0000 mon.a (mon.0) 896 : cluster [DBG] mgrmap e22: y(active, since 7m), standbys: x 2023-12-16T07:42:51.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:51 smithi169 bash[25083]: audit 2023-12-16T07:42:50.550887+0000 mon.a (mon.0) 897 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:42:51.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:51 smithi169 bash[25083]: audit 2023-12-16T07:42:50.563934+0000 mon.a (mon.0) 898 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:42:52.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:52 smithi169 bash[45188]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:42:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:42:52.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:52 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:52.364Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=7 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:42:53.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:42:53] "GET /metrics HTTP/1.1" 200 214848 "" "Prometheus/2.33.4" 2023-12-16T07:42:53.792 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[49264]: level=error ts=2023-12-16T07:42:53.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:42:53.792 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:53.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:42:53.792 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:53.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2023-12-16T07:42:54.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25282]: ignoring --setuser ceph since I am not root 2023-12-16T07:42:54.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25282]: ignoring --setgroup ceph since I am not root 2023-12-16T07:42:54.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25282]: debug 2023-12-16T07:42:54.007+0000 7f7377412700 1 -- 172.21.15.7:0/1718384915 <== mon.2 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5650b37e6340 con 0x5650b4562800 2023-12-16T07:42:54.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[30436]: cluster 2023-12-16T07:42:52.393789+0000 mgr.y (mgr.24367) 313 : cluster [DBG] pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:54.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[30436]: audit 2023-12-16T07:42:52.948030+0000 mon.a (mon.0) 899 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:54.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[30436]: audit 2023-12-16T07:42:52.957902+0000 mon.a (mon.0) 900 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:54.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[30436]: audit 2023-12-16T07:42:52.967951+0000 mon.a (mon.0) 901 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:42:54.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[30436]: cephadm 2023-12-16T07:42:52.971227+0000 mgr.y (mgr.24367) 314 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-12-16T07:42:54.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[30436]: cephadm 2023-12-16T07:42:52.974826+0000 mgr.y (mgr.24367) 315 : cephadm [INF] Failing over to other MGR 2023-12-16T07:42:54.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[30436]: audit 2023-12-16T07:42:52.975217+0000 mon.a (mon.0) 902 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2023-12-16T07:42:54.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[30436]: cluster 2023-12-16T07:42:52.983823+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2023-12-16T07:42:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[25009]: cluster 2023-12-16T07:42:52.393789+0000 mgr.y (mgr.24367) 313 : cluster [DBG] pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[25009]: audit 2023-12-16T07:42:52.948030+0000 mon.a (mon.0) 899 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[25009]: audit 2023-12-16T07:42:52.957902+0000 mon.a (mon.0) 900 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[25009]: audit 2023-12-16T07:42:52.967951+0000 mon.a (mon.0) 901 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:42:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[25009]: cephadm 2023-12-16T07:42:52.971227+0000 mgr.y (mgr.24367) 314 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-12-16T07:42:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[25009]: cephadm 2023-12-16T07:42:52.974826+0000 mgr.y (mgr.24367) 315 : cephadm [INF] Failing over to other MGR 2023-12-16T07:42:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[25009]: audit 2023-12-16T07:42:52.975217+0000 mon.a (mon.0) 902 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2023-12-16T07:42:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:53 smithi007 bash[25009]: cluster 2023-12-16T07:42:52.983823+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2023-12-16T07:42:54.228 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:53 smithi169 bash[25083]: cluster 2023-12-16T07:42:52.393789+0000 mgr.y (mgr.24367) 313 : cluster [DBG] pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:42:54.229 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:53 smithi169 bash[25083]: audit 2023-12-16T07:42:52.948030+0000 mon.a (mon.0) 899 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:54.229 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:53 smithi169 bash[25083]: audit 2023-12-16T07:42:52.957902+0000 mon.a (mon.0) 900 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' 2023-12-16T07:42:54.229 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:53 smithi169 bash[25083]: audit 2023-12-16T07:42:52.967951+0000 mon.a (mon.0) 901 : audit [DBG] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:42:54.229 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:53 smithi169 bash[25083]: cephadm 2023-12-16T07:42:52.971227+0000 mgr.y (mgr.24367) 314 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.y) 2023-12-16T07:42:54.229 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:53 smithi169 bash[25083]: cephadm 2023-12-16T07:42:52.974826+0000 mgr.y (mgr.24367) 315 : cephadm [INF] Failing over to other MGR 2023-12-16T07:42:54.229 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:53 smithi169 bash[25083]: audit 2023-12-16T07:42:52.975217+0000 mon.a (mon.0) 902 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2023-12-16T07:42:54.229 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:53 smithi169 bash[25083]: cluster 2023-12-16T07:42:52.983823+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2023-12-16T07:42:54.230 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:53 smithi169 bash[45188]: [16/Dec/2023:07:42:53] ENGINE Bus STOPPING 2023-12-16T07:42:54.230 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:53 smithi169 bash[45188]: [16/Dec/2023:07:42:53] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2023-12-16T07:42:54.230 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:53 smithi169 bash[45188]: [16/Dec/2023:07:42:53] ENGINE Bus STOPPED 2023-12-16T07:42:54.230 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[45188]: [16/Dec/2023:07:42:54] ENGINE Bus STARTING 2023-12-16T07:42:54.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25282]: debug 2023-12-16T07:42:54.171+0000 7f737fe6e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T07:42:54.567 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:54.147Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=2 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused" 2023-12-16T07:42:54.567 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:54.157Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=2 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:42:54.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[45188]: [16/Dec/2023:07:42:54] ENGINE Serving on http://:::9283 2023-12-16T07:42:54.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[45188]: [16/Dec/2023:07:42:54] ENGINE Bus STARTED 2023-12-16T07:42:54.962 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25282]: debug 2023-12-16T07:42:54.875+0000 7f737fe6e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T07:42:54.962 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[49264]: level=warn ts=2023-12-16T07:42:54.672Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=3 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:42:55.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[25282]: debug 2023-12-16T07:42:55.007+0000 7f737fe6e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T07:42:55.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[25282]: debug 2023-12-16T07:42:55.175+0000 7f737fe6e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:42:55.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.965595+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2023-12-16T07:42:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: cluster 2023-12-16T07:42:53.965720+0000 mon.a (mon.0) 905 : cluster [DBG] mgrmap e23: x(active, starting, since 0.98919s) 2023-12-16T07:42:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.970920+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:42:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.971278+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:42:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.971603+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:42:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.972768+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:42:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.975461+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:42:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.975649+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:42:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.976179+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:42:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.976480+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:42:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.976760+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:42:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.976982+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:42:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.977342+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:42:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.977638+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:42:55.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.977992+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:42:55.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.978324+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:42:55.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:53.978749+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:42:55.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: cluster 2023-12-16T07:42:54.022661+0000 mon.a (mon.0) 906 : cluster [INF] Manager daemon x is now available 2023-12-16T07:42:55.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:54.077643+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: cephadm 2023-12-16T07:42:54.079932+0000 mgr.x (mgr.24968) 1 : cephadm [INF] Queued rgw.foo for migration 2023-12-16T07:42:55.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:54.085568+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: cephadm 2023-12-16T07:42:54.087999+0000 mgr.x (mgr.24968) 2 : cephadm [INF] Queued rgw.smpl for migration 2023-12-16T07:42:55.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: cephadm 2023-12-16T07:42:54.089103+0000 mgr.x (mgr.24968) 3 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2023-12-16T07:42:55.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: cephadm 2023-12-16T07:42:54.089268+0000 mgr.x (mgr.24968) 4 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2023-12-16T07:42:55.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:54.097056+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:54.113923+0000 mon.b (mon.2) 44 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:42:55.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:54.162271+0000 mon.b (mon.2) 45 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:42:55.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:54.173751+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:42:55.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:54.174355+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:42:55.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:54.224793+0000 mon.a (mon.0) 911 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-12-16T07:42:55.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:54.225398+0000 mon.b (mon.2) 47 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-12-16T07:42:55.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:54.299708+0000 mon.a (mon.0) 912 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: audit 2023-12-16T07:42:54.308580+0000 mon.a (mon.0) 913 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.321 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: cephadm 2023-12-16T07:42:54.635106+0000 mgr.x (mgr.24968) 5 : cephadm [INF] [16/Dec/2023:07:42:54] ENGINE Bus STARTING 2023-12-16T07:42:55.321 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: cephadm 2023-12-16T07:42:54.667686+0000 mgr.x (mgr.24968) 6 : cephadm [INF] Deploying cephadm binary to smithi169 2023-12-16T07:42:55.321 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[25009]: cephadm 2023-12-16T07:42:54.672300+0000 mgr.x (mgr.24968) 7 : cephadm [INF] Deploying cephadm binary to smithi007 2023-12-16T07:42:55.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.965595+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2023-12-16T07:42:55.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: cluster 2023-12-16T07:42:53.965720+0000 mon.a (mon.0) 905 : cluster [DBG] mgrmap e23: x(active, starting, since 0.98919s) 2023-12-16T07:42:55.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.970920+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:42:55.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.971278+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:42:55.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.971603+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:42:55.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.972768+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:42:55.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.975461+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:42:55.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.975649+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:42:55.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.976179+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:42:55.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.976480+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:42:55.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.976760+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:42:55.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.976982+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:42:55.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.977342+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:42:55.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.977638+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:42:55.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.977992+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:42:55.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.978324+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:42:55.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:53.978749+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:42:55.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: cluster 2023-12-16T07:42:54.022661+0000 mon.a (mon.0) 906 : cluster [INF] Manager daemon x is now available 2023-12-16T07:42:55.323 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:54.077643+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.323 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: cephadm 2023-12-16T07:42:54.079932+0000 mgr.x (mgr.24968) 1 : cephadm [INF] Queued rgw.foo for migration 2023-12-16T07:42:55.323 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:54.085568+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.323 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: cephadm 2023-12-16T07:42:54.087999+0000 mgr.x (mgr.24968) 2 : cephadm [INF] Queued rgw.smpl for migration 2023-12-16T07:42:55.323 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: cephadm 2023-12-16T07:42:54.089103+0000 mgr.x (mgr.24968) 3 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2023-12-16T07:42:55.323 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: cephadm 2023-12-16T07:42:54.089268+0000 mgr.x (mgr.24968) 4 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2023-12-16T07:42:55.323 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:54.097056+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.323 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:54.113923+0000 mon.b (mon.2) 44 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:42:55.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:54.162271+0000 mon.b (mon.2) 45 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:42:55.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:54.173751+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:42:55.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:54.174355+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:42:55.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:54.224793+0000 mon.a (mon.0) 911 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-12-16T07:42:55.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:54.225398+0000 mon.b (mon.2) 47 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-12-16T07:42:55.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:54.299708+0000 mon.a (mon.0) 912 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: audit 2023-12-16T07:42:54.308580+0000 mon.a (mon.0) 913 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: cephadm 2023-12-16T07:42:54.635106+0000 mgr.x (mgr.24968) 5 : cephadm [INF] [16/Dec/2023:07:42:54] ENGINE Bus STARTING 2023-12-16T07:42:55.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: cephadm 2023-12-16T07:42:54.667686+0000 mgr.x (mgr.24968) 6 : cephadm [INF] Deploying cephadm binary to smithi169 2023-12-16T07:42:55.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:54 smithi007 bash[30436]: cephadm 2023-12-16T07:42:54.672300+0000 mgr.x (mgr.24968) 7 : cephadm [INF] Deploying cephadm binary to smithi007 2023-12-16T07:42:55.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.965595+0000 mon.a (mon.0) 904 : audit [INF] from='mgr.24367 172.21.15.7:0/842464694' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2023-12-16T07:42:55.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: cluster 2023-12-16T07:42:53.965720+0000 mon.a (mon.0) 905 : cluster [DBG] mgrmap e23: x(active, starting, since 0.98919s) 2023-12-16T07:42:55.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.970920+0000 mon.b (mon.2) 29 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:42:55.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.971278+0000 mon.b (mon.2) 30 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:42:55.435 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.971603+0000 mon.b (mon.2) 31 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:42:55.435 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.972768+0000 mon.b (mon.2) 32 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:42:55.435 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.975461+0000 mon.b (mon.2) 33 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:42:55.435 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.975649+0000 mon.b (mon.2) 34 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:42:55.435 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.976179+0000 mon.b (mon.2) 35 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:42:55.435 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.976480+0000 mon.b (mon.2) 36 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:42:55.436 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.976760+0000 mon.b (mon.2) 37 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:42:55.436 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.976982+0000 mon.b (mon.2) 38 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:42:55.436 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.977342+0000 mon.b (mon.2) 39 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:42:55.436 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.977638+0000 mon.b (mon.2) 40 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:42:55.436 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.977992+0000 mon.b (mon.2) 41 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:42:55.436 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.978324+0000 mon.b (mon.2) 42 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:42:55.437 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:53.978749+0000 mon.b (mon.2) 43 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:42:55.437 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: cluster 2023-12-16T07:42:54.022661+0000 mon.a (mon.0) 906 : cluster [INF] Manager daemon x is now available 2023-12-16T07:42:55.437 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:54.077643+0000 mon.a (mon.0) 907 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.437 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: cephadm 2023-12-16T07:42:54.079932+0000 mgr.x (mgr.24968) 1 : cephadm [INF] Queued rgw.foo for migration 2023-12-16T07:42:55.437 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:54.085568+0000 mon.a (mon.0) 908 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.438 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: cephadm 2023-12-16T07:42:54.087999+0000 mgr.x (mgr.24968) 2 : cephadm [INF] Queued rgw.smpl for migration 2023-12-16T07:42:55.438 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: cephadm 2023-12-16T07:42:54.089103+0000 mgr.x (mgr.24968) 3 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2023-12-16T07:42:55.438 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: cephadm 2023-12-16T07:42:54.089268+0000 mgr.x (mgr.24968) 4 : cephadm [INF] No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2023-12-16T07:42:55.438 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:54.097056+0000 mon.a (mon.0) 909 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.438 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:54.113923+0000 mon.b (mon.2) 44 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:42:55.438 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:54.162271+0000 mon.b (mon.2) 45 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:42:55.439 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:54.173751+0000 mon.a (mon.0) 910 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:42:55.439 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:54.174355+0000 mon.b (mon.2) 46 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:42:55.439 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:54.224793+0000 mon.a (mon.0) 911 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-12-16T07:42:55.439 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:54.225398+0000 mon.b (mon.2) 47 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-12-16T07:42:55.439 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:54.299708+0000 mon.a (mon.0) 912 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.439 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: audit 2023-12-16T07:42:54.308580+0000 mon.a (mon.0) 913 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:42:55.440 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: cephadm 2023-12-16T07:42:54.635106+0000 mgr.x (mgr.24968) 5 : cephadm [INF] [16/Dec/2023:07:42:54] ENGINE Bus STARTING 2023-12-16T07:42:55.440 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: cephadm 2023-12-16T07:42:54.667686+0000 mgr.x (mgr.24968) 6 : cephadm [INF] Deploying cephadm binary to smithi169 2023-12-16T07:42:55.440 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:54 smithi169 bash[25083]: cephadm 2023-12-16T07:42:54.672300+0000 mgr.x (mgr.24968) 7 : cephadm [INF] Deploying cephadm binary to smithi007 2023-12-16T07:42:55.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[25282]: debug 2023-12-16T07:42:55.515+0000 7f737fe6e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T07:42:55.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[25282]: debug 2023-12-16T07:42:55.611+0000 7f737fe6e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T07:42:56.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[25282]: debug 2023-12-16T07:42:55.823+0000 7f737fe6e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T07:42:56.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[25009]: cephadm 2023-12-16T07:42:54.737724+0000 mgr.x (mgr.24968) 8 : cephadm [INF] [16/Dec/2023:07:42:54] ENGINE Serving on http://172.21.15.169:8765 2023-12-16T07:42:56.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[25009]: cephadm 2023-12-16T07:42:54.914613+0000 mgr.x (mgr.24968) 9 : cephadm [INF] [16/Dec/2023:07:42:54] ENGINE Serving on https://172.21.15.169:7150 2023-12-16T07:42:56.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[25009]: cephadm 2023-12-16T07:42:54.914962+0000 mgr.x (mgr.24968) 10 : cephadm [INF] [16/Dec/2023:07:42:54] ENGINE Bus STARTED 2023-12-16T07:42:56.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[25009]: cluster 2023-12-16T07:42:54.978635+0000 mon.a (mon.0) 914 : cluster [DBG] mgrmap e24: x(active, since 2s) 2023-12-16T07:42:56.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[25009]: cluster 2023-12-16T07:42:55.025723+0000 mgr.x (mgr.24968) 11 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:42:56.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[30436]: cephadm 2023-12-16T07:42:54.737724+0000 mgr.x (mgr.24968) 8 : cephadm [INF] [16/Dec/2023:07:42:54] ENGINE Serving on http://172.21.15.169:8765 2023-12-16T07:42:56.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[30436]: cephadm 2023-12-16T07:42:54.914613+0000 mgr.x (mgr.24968) 9 : cephadm [INF] [16/Dec/2023:07:42:54] ENGINE Serving on https://172.21.15.169:7150 2023-12-16T07:42:56.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[30436]: cephadm 2023-12-16T07:42:54.914962+0000 mgr.x (mgr.24968) 10 : cephadm [INF] [16/Dec/2023:07:42:54] ENGINE Bus STARTED 2023-12-16T07:42:56.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[30436]: cluster 2023-12-16T07:42:54.978635+0000 mon.a (mon.0) 914 : cluster [DBG] mgrmap e24: x(active, since 2s) 2023-12-16T07:42:56.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:55 smithi007 bash[30436]: cluster 2023-12-16T07:42:55.025723+0000 mgr.x (mgr.24968) 11 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:42:56.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:55 smithi169 bash[25083]: cephadm 2023-12-16T07:42:54.737724+0000 mgr.x (mgr.24968) 8 : cephadm [INF] [16/Dec/2023:07:42:54] ENGINE Serving on http://172.21.15.169:8765 2023-12-16T07:42:56.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:55 smithi169 bash[25083]: cephadm 2023-12-16T07:42:54.914613+0000 mgr.x (mgr.24968) 9 : cephadm [INF] [16/Dec/2023:07:42:54] ENGINE Serving on https://172.21.15.169:7150 2023-12-16T07:42:56.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:55 smithi169 bash[25083]: cephadm 2023-12-16T07:42:54.914962+0000 mgr.x (mgr.24968) 10 : cephadm [INF] [16/Dec/2023:07:42:54] ENGINE Bus STARTED 2023-12-16T07:42:56.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:55 smithi169 bash[25083]: cluster 2023-12-16T07:42:54.978635+0000 mon.a (mon.0) 914 : cluster [DBG] mgrmap e24: x(active, since 2s) 2023-12-16T07:42:56.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:55 smithi169 bash[25083]: cluster 2023-12-16T07:42:55.025723+0000 mgr.x (mgr.24968) 11 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:42:56.999 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:56 smithi007 bash[25282]: debug 2023-12-16T07:42:56.715+0000 7f737fe6e000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T07:42:56.999 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:56 smithi007 bash[25282]: debug 2023-12-16T07:42:56.827+0000 7f737fe6e000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T07:42:57.000 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:56 smithi007 bash[25009]: cluster 2023-12-16T07:42:55.975918+0000 mgr.x (mgr.24968) 12 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:42:57.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:57 smithi007 bash[25282]: debug 2023-12-16T07:42:57.039+0000 7f737fe6e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T07:42:57.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:56 smithi007 bash[30436]: cluster 2023-12-16T07:42:55.975918+0000 mgr.x (mgr.24968) 12 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:42:57.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:56 smithi169 bash[25083]: cluster 2023-12-16T07:42:55.975918+0000 mgr.x (mgr.24968) 12 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:42:57.751 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:57 smithi007 bash[25282]: debug 2023-12-16T07:42:57.403+0000 7f737fe6e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T07:42:58.001 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:57 smithi007 bash[25282]: debug 2023-12-16T07:42:57.743+0000 7f737fe6e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T07:42:58.003 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:58 smithi007 bash[30436]: cluster 2023-12-16T07:42:56.991545+0000 mon.a (mon.0) 915 : cluster [DBG] mgrmap e25: x(active, since 4s) 2023-12-16T07:42:58.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:58 smithi007 bash[25009]: cluster 2023-12-16T07:42:56.991545+0000 mon.a (mon.0) 915 : cluster [DBG] mgrmap e25: x(active, since 4s) 2023-12-16T07:42:58.318 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:58 smithi007 bash[25282]: debug 2023-12-16T07:42:58.131+0000 7f737fe6e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T07:42:58.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:58 smithi169 bash[25083]: cluster 2023-12-16T07:42:56.991545+0000 mon.a (mon.0) 915 : cluster [DBG] mgrmap e25: x(active, since 4s) 2023-12-16T07:42:58.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:58 smithi007 bash[25282]: debug 2023-12-16T07:42:58.415+0000 7f737fe6e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:42:59.422 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:59 smithi007 bash[25282]: debug 2023-12-16T07:42:59.138+0000 7f737fe6e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T07:42:59.422 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:59 smithi007 bash[25282]: debug 2023-12-16T07:42:59.250+0000 7f737fe6e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T07:42:59.422 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:59 smithi007 bash[30436]: cluster 2023-12-16T07:42:57.976686+0000 mgr.x (mgr.24968) 13 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:42:59.422 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:42:59 smithi007 bash[30436]: cluster 2023-12-16T07:42:59.011314+0000 mon.a (mon.0) 916 : cluster [DBG] mgrmap e26: x(active, since 6s) 2023-12-16T07:42:59.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:59 smithi007 bash[25009]: cluster 2023-12-16T07:42:57.976686+0000 mgr.x (mgr.24968) 13 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:42:59.423 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:42:59 smithi007 bash[25009]: cluster 2023-12-16T07:42:59.011314+0000 mon.a (mon.0) 916 : cluster [DBG] mgrmap e26: x(active, since 6s) 2023-12-16T07:42:59.683 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:59 smithi007 bash[25282]: debug 2023-12-16T07:42:59.414+0000 7f737fe6e000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T07:42:59.683 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:59 smithi007 bash[25282]: debug 2023-12-16T07:42:59.530+0000 7f737fe6e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T07:42:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:59 smithi169 bash[25083]: cluster 2023-12-16T07:42:57.976686+0000 mgr.x (mgr.24968) 13 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:42:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:42:59 smithi169 bash[25083]: cluster 2023-12-16T07:42:59.011314+0000 mon.a (mon.0) 916 : cluster [DBG] mgrmap e26: x(active, since 6s) 2023-12-16T07:42:59.984 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:59 smithi007 bash[25282]: debug 2023-12-16T07:42:59.674+0000 7f737fe6e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T07:43:00.306 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:42:59 smithi007 bash[25282]: debug 2023-12-16T07:42:59.974+0000 7f737fe6e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T07:43:00.306 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:43:00 smithi007 bash[25282]: debug 2023-12-16T07:43:00.102+0000 7f737fe6e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T07:43:00.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:43:00 smithi007 bash[25282]: debug 2023-12-16T07:43:00.298+0000 7f737fe6e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T07:43:00.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:43:00 smithi007 bash[25282]: debug 2023-12-16T07:43:00.422+0000 7f737fe6e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T07:43:01.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:01 smithi007 bash[25009]: cluster 2023-12-16T07:42:59.977523+0000 mgr.x (mgr.24968) 14 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:43:01.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:01 smithi007 bash[30436]: cluster 2023-12-16T07:42:59.977523+0000 mgr.x (mgr.24968) 14 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:43:01.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:43:01 smithi007 bash[25282]: debug 2023-12-16T07:43:01.318+0000 7f737fe6e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T07:43:01.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:43:01 smithi007 bash[25282]: [16/Dec/2023:07:43:01] ENGINE Bus STARTING 2023-12-16T07:43:01.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:43:01 smithi007 bash[25282]: CherryPy Checker: 2023-12-16T07:43:01.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:43:01 smithi007 bash[25282]: The Application mounted at '' has an empty config. 2023-12-16T07:43:01.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:43:01 smithi007 bash[25282]: [16/Dec/2023:07:43:01] ENGINE Serving on http://:::9283 2023-12-16T07:43:01.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:43:01 smithi007 bash[25282]: [16/Dec/2023:07:43:01] ENGINE Bus STARTED 2023-12-16T07:43:01.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:01 smithi169 bash[25083]: cluster 2023-12-16T07:42:59.977523+0000 mgr.x (mgr.24968) 14 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:43:02.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:02 smithi007 bash[25009]: cluster 2023-12-16T07:43:01.344993+0000 mon.a (mon.0) 917 : cluster [DBG] Standby manager daemon y started 2023-12-16T07:43:02.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:02 smithi007 bash[25009]: audit 2023-12-16T07:43:01.350393+0000 mon.a (mon.0) 918 : audit [DBG] from='mgr.? 172.21.15.7:0/3956629413' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-12-16T07:43:02.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:02 smithi007 bash[25009]: audit 2023-12-16T07:43:01.353938+0000 mon.a (mon.0) 919 : audit [DBG] from='mgr.? 172.21.15.7:0/3956629413' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:43:02.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:02 smithi007 bash[25009]: audit 2023-12-16T07:43:01.357285+0000 mon.a (mon.0) 920 : audit [DBG] from='mgr.? 172.21.15.7:0/3956629413' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-12-16T07:43:02.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:02 smithi007 bash[25009]: audit 2023-12-16T07:43:01.357922+0000 mon.a (mon.0) 921 : audit [DBG] from='mgr.? 172.21.15.7:0/3956629413' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:43:02.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:02 smithi007 bash[30436]: cluster 2023-12-16T07:43:01.344993+0000 mon.a (mon.0) 917 : cluster [DBG] Standby manager daemon y started 2023-12-16T07:43:02.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:02 smithi007 bash[30436]: audit 2023-12-16T07:43:01.350393+0000 mon.a (mon.0) 918 : audit [DBG] from='mgr.? 172.21.15.7:0/3956629413' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-12-16T07:43:02.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:02 smithi007 bash[30436]: audit 2023-12-16T07:43:01.353938+0000 mon.a (mon.0) 919 : audit [DBG] from='mgr.? 172.21.15.7:0/3956629413' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:43:02.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:02 smithi007 bash[30436]: audit 2023-12-16T07:43:01.357285+0000 mon.a (mon.0) 920 : audit [DBG] from='mgr.? 172.21.15.7:0/3956629413' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-12-16T07:43:02.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:02 smithi007 bash[30436]: audit 2023-12-16T07:43:01.357922+0000 mon.a (mon.0) 921 : audit [DBG] from='mgr.? 172.21.15.7:0/3956629413' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:43:02.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:43:02 smithi169 bash[45188]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:43:02] "GET /metrics HTTP/1.1" 200 35139 "" "Prometheus/2.33.4" 2023-12-16T07:43:02.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:02 smithi169 bash[25083]: cluster 2023-12-16T07:43:01.344993+0000 mon.a (mon.0) 917 : cluster [DBG] Standby manager daemon y started 2023-12-16T07:43:02.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:02 smithi169 bash[25083]: audit 2023-12-16T07:43:01.350393+0000 mon.a (mon.0) 918 : audit [DBG] from='mgr.? 172.21.15.7:0/3956629413' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-12-16T07:43:02.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:02 smithi169 bash[25083]: audit 2023-12-16T07:43:01.353938+0000 mon.a (mon.0) 919 : audit [DBG] from='mgr.? 172.21.15.7:0/3956629413' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:43:02.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:02 smithi169 bash[25083]: audit 2023-12-16T07:43:01.357285+0000 mon.a (mon.0) 920 : audit [DBG] from='mgr.? 172.21.15.7:0/3956629413' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-12-16T07:43:02.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:02 smithi169 bash[25083]: audit 2023-12-16T07:43:01.357922+0000 mon.a (mon.0) 921 : audit [DBG] from='mgr.? 172.21.15.7:0/3956629413' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:43:03.520 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:03 smithi007 bash[30436]: cluster 2023-12-16T07:43:01.979223+0000 mgr.x (mgr.24968) 15 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2023-12-16T07:43:03.520 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:03 smithi007 bash[30436]: cluster 2023-12-16T07:43:02.251887+0000 mon.a (mon.0) 922 : cluster [DBG] mgrmap e27: x(active, since 9s), standbys: y 2023-12-16T07:43:03.520 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:03 smithi007 bash[30436]: audit 2023-12-16T07:43:02.260082+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:43:03.520 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:03 smithi007 bash[25009]: cluster 2023-12-16T07:43:01.979223+0000 mgr.x (mgr.24968) 15 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2023-12-16T07:43:03.520 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:03 smithi007 bash[25009]: cluster 2023-12-16T07:43:02.251887+0000 mon.a (mon.0) 922 : cluster [DBG] mgrmap e27: x(active, since 9s), standbys: y 2023-12-16T07:43:03.521 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:03 smithi007 bash[25009]: audit 2023-12-16T07:43:02.260082+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:43:03.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:03 smithi169 bash[25083]: cluster 2023-12-16T07:43:01.979223+0000 mgr.x (mgr.24968) 15 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2023-12-16T07:43:03.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:03 smithi169 bash[25083]: cluster 2023-12-16T07:43:02.251887+0000 mon.a (mon.0) 922 : cluster [DBG] mgrmap e27: x(active, since 9s), standbys: y 2023-12-16T07:43:03.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:03 smithi169 bash[25083]: audit 2023-12-16T07:43:02.260082+0000 mon.b (mon.2) 48 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:43:03.800 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:43:03 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:43:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:43:03.800 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:03 smithi007 bash[49264]: level=error ts=2023-12-16T07:43:03.517Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:43:03.800 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:03 smithi007 bash[49264]: level=warn ts=2023-12-16T07:43:03.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:43:03.800 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:03 smithi007 bash[49264]: level=warn ts=2023-12-16T07:43:03.524Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2023-12-16T07:43:05.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:05 smithi007 bash[30436]: audit 2023-12-16T07:43:03.798645+0000 mgr.x (mgr.24968) 16 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:05.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:05 smithi007 bash[30436]: cluster 2023-12-16T07:43:03.980016+0000 mgr.x (mgr.24968) 17 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2023-12-16T07:43:05.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:05 smithi007 bash[25009]: audit 2023-12-16T07:43:03.798645+0000 mgr.x (mgr.24968) 16 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:05.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:05 smithi007 bash[25009]: cluster 2023-12-16T07:43:03.980016+0000 mgr.x (mgr.24968) 17 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2023-12-16T07:43:05.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:05 smithi169 bash[25083]: audit 2023-12-16T07:43:03.798645+0000 mgr.x (mgr.24968) 16 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:05.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:05 smithi169 bash[25083]: cluster 2023-12-16T07:43:03.980016+0000 mgr.x (mgr.24968) 17 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2023-12-16T07:43:07.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:07 smithi007 bash[30436]: cluster 2023-12-16T07:43:05.981808+0000 mgr.x (mgr.24968) 18 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2023-12-16T07:43:07.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:07 smithi007 bash[25009]: cluster 2023-12-16T07:43:05.981808+0000 mgr.x (mgr.24968) 18 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2023-12-16T07:43:07.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:07 smithi169 bash[25083]: cluster 2023-12-16T07:43:05.981808+0000 mgr.x (mgr.24968) 18 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2023-12-16T07:43:09.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:09 smithi007 bash[25009]: cluster 2023-12-16T07:43:07.982689+0000 mgr.x (mgr.24968) 19 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:43:09.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:09 smithi007 bash[25009]: audit 2023-12-16T07:43:08.476861+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:09.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:09 smithi007 bash[25009]: audit 2023-12-16T07:43:08.488797+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:09.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:09 smithi007 bash[25009]: audit 2023-12-16T07:43:08.507026+0000 mon.a (mon.0) 925 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:09.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:09 smithi007 bash[25009]: audit 2023-12-16T07:43:08.520480+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:09.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:09 smithi007 bash[25009]: audit 2023-12-16T07:43:09.164729+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:43:09.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:09 smithi007 bash[30436]: cluster 2023-12-16T07:43:07.982689+0000 mgr.x (mgr.24968) 19 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:43:09.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:09 smithi007 bash[30436]: audit 2023-12-16T07:43:08.476861+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:09.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:09 smithi007 bash[30436]: audit 2023-12-16T07:43:08.488797+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:09.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:09 smithi007 bash[30436]: audit 2023-12-16T07:43:08.507026+0000 mon.a (mon.0) 925 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:09.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:09 smithi007 bash[30436]: audit 2023-12-16T07:43:08.520480+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:09.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:09 smithi007 bash[30436]: audit 2023-12-16T07:43:09.164729+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:43:09.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:09 smithi169 bash[25083]: cluster 2023-12-16T07:43:07.982689+0000 mgr.x (mgr.24968) 19 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:43:09.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:09 smithi169 bash[25083]: audit 2023-12-16T07:43:08.476861+0000 mon.a (mon.0) 923 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:09.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:09 smithi169 bash[25083]: audit 2023-12-16T07:43:08.488797+0000 mon.a (mon.0) 924 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:09.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:09 smithi169 bash[25083]: audit 2023-12-16T07:43:08.507026+0000 mon.a (mon.0) 925 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:09.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:09 smithi169 bash[25083]: audit 2023-12-16T07:43:08.520480+0000 mon.a (mon.0) 926 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:09.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:09 smithi169 bash[25083]: audit 2023-12-16T07:43:09.164729+0000 mon.b (mon.2) 49 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:43:10.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:10 smithi169 bash[25083]: audit 2023-12-16T07:43:09.635793+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:10.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:10 smithi169 bash[25083]: audit 2023-12-16T07:43:09.648873+0000 mon.a (mon.0) 928 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:10.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:10 smithi169 bash[25083]: audit 2023-12-16T07:43:09.655700+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:10.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:10 smithi169 bash[25083]: audit 2023-12-16T07:43:09.657963+0000 mon.a (mon.0) 930 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:10.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:10 smithi169 bash[25083]: audit 2023-12-16T07:43:09.658741+0000 mon.b (mon.2) 50 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:10.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:10 smithi169 bash[25083]: audit 2023-12-16T07:43:09.663465+0000 mon.b (mon.2) 51 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:10.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:10 smithi169 bash[25083]: audit 2023-12-16T07:43:09.663522+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:10.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:10 smithi169 bash[25083]: audit 2023-12-16T07:43:09.666907+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:10.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:10 smithi169 bash[25083]: audit 2023-12-16T07:43:09.671948+0000 mon.a (mon.0) 933 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:10.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:10 smithi169 bash[25083]: audit 2023-12-16T07:43:09.672586+0000 mon.b (mon.2) 52 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:10.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:10 smithi169 bash[25083]: audit 2023-12-16T07:43:09.675176+0000 mon.a (mon.0) 934 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:10.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:10 smithi169 bash[25083]: audit 2023-12-16T07:43:09.675875+0000 mon.b (mon.2) 53 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:10.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:10 smithi169 bash[25083]: cephadm 2023-12-16T07:43:09.678727+0000 mgr.x (mgr.24968) 20 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 2782M 2023-12-16T07:43:10.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:10 smithi169 bash[25083]: audit 2023-12-16T07:43:09.685673+0000 mon.a (mon.0) 935 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:11.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[25009]: audit 2023-12-16T07:43:09.635793+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:11.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[25009]: audit 2023-12-16T07:43:09.648873+0000 mon.a (mon.0) 928 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:11.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[25009]: audit 2023-12-16T07:43:09.655700+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:11.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[25009]: audit 2023-12-16T07:43:09.657963+0000 mon.a (mon.0) 930 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[25009]: audit 2023-12-16T07:43:09.658741+0000 mon.b (mon.2) 50 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[25009]: audit 2023-12-16T07:43:09.663465+0000 mon.b (mon.2) 51 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[25009]: audit 2023-12-16T07:43:09.663522+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[25009]: audit 2023-12-16T07:43:09.666907+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:11.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[25009]: audit 2023-12-16T07:43:09.671948+0000 mon.a (mon.0) 933 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[25009]: audit 2023-12-16T07:43:09.672586+0000 mon.b (mon.2) 52 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[25009]: audit 2023-12-16T07:43:09.675176+0000 mon.a (mon.0) 934 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[25009]: audit 2023-12-16T07:43:09.675875+0000 mon.b (mon.2) 53 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[25009]: cephadm 2023-12-16T07:43:09.678727+0000 mgr.x (mgr.24968) 20 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 2782M 2023-12-16T07:43:11.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[25009]: audit 2023-12-16T07:43:09.685673+0000 mon.a (mon.0) 935 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:11.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[30436]: audit 2023-12-16T07:43:09.635793+0000 mon.a (mon.0) 927 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:11.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[30436]: audit 2023-12-16T07:43:09.648873+0000 mon.a (mon.0) 928 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:11.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[30436]: audit 2023-12-16T07:43:09.655700+0000 mon.a (mon.0) 929 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:11.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[30436]: audit 2023-12-16T07:43:09.657963+0000 mon.a (mon.0) 930 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[30436]: audit 2023-12-16T07:43:09.658741+0000 mon.b (mon.2) 50 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[30436]: audit 2023-12-16T07:43:09.663465+0000 mon.b (mon.2) 51 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[30436]: audit 2023-12-16T07:43:09.663522+0000 mon.a (mon.0) 931 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[30436]: audit 2023-12-16T07:43:09.666907+0000 mon.a (mon.0) 932 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:11.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[30436]: audit 2023-12-16T07:43:09.671948+0000 mon.a (mon.0) 933 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[30436]: audit 2023-12-16T07:43:09.672586+0000 mon.b (mon.2) 52 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[30436]: audit 2023-12-16T07:43:09.675176+0000 mon.a (mon.0) 934 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[30436]: audit 2023-12-16T07:43:09.675875+0000 mon.b (mon.2) 53 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:11.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[30436]: cephadm 2023-12-16T07:43:09.678727+0000 mgr.x (mgr.24968) 20 : cephadm [INF] Adjusting osd_memory_target on smithi007 to 2782M 2023-12-16T07:43:11.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:10 smithi007 bash[30436]: audit 2023-12-16T07:43:09.685673+0000 mon.a (mon.0) 935 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:12.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:11 smithi007 bash[25009]: cluster 2023-12-16T07:43:09.983581+0000 mgr.x (mgr.24968) 21 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:43:12.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:11 smithi007 bash[30436]: cluster 2023-12-16T07:43:09.983581+0000 mgr.x (mgr.24968) 21 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:43:12.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:11 smithi169 bash[25083]: cluster 2023-12-16T07:43:09.983581+0000 mgr.x (mgr.24968) 21 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:43:12.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:43:12 smithi169 bash[45188]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:43:12] "GET /metrics HTTP/1.1" 200 38136 "" "Prometheus/2.33.4" 2023-12-16T07:43:12.684 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:12 smithi169 bash[41600]: ts=2023-12-16T07:43:12.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.7:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.169:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:43:13.810 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:13 smithi007 bash[25009]: cluster 2023-12-16T07:43:11.985312+0000 mgr.x (mgr.24968) 22 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:43:13.810 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:43:13 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:43:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:43:13.810 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:13 smithi007 bash[30436]: cluster 2023-12-16T07:43:11.985312+0000 mgr.x (mgr.24968) 22 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:43:13.811 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:13 smithi007 bash[49264]: level=error ts=2023-12-16T07:43:13.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2023-12-16T07:43:13.811 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:13 smithi007 bash[49264]: level=warn ts=2023-12-16T07:43:13.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2023-12-16T07:43:13.811 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:13 smithi007 bash[49264]: level=warn ts=2023-12-16T07:43:13.524Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:43:14.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:13 smithi169 bash[25083]: cluster 2023-12-16T07:43:11.985312+0000 mgr.x (mgr.24968) 22 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:43:15.647 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:43:16.052 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:15 smithi169 bash[25083]: audit 2023-12-16T07:43:13.808832+0000 mgr.x (mgr.24968) 23 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:16.052 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:15 smithi169 bash[25083]: cluster 2023-12-16T07:43:13.986086+0000 mgr.x (mgr.24968) 24 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:16.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:15 smithi007 bash[25009]: audit 2023-12-16T07:43:13.808832+0000 mgr.x (mgr.24968) 23 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:16.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:15 smithi007 bash[25009]: cluster 2023-12-16T07:43:13.986086+0000 mgr.x (mgr.24968) 24 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:16.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:15 smithi007 bash[30436]: audit 2023-12-16T07:43:13.808832+0000 mgr.x (mgr.24968) 23 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:16.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:15 smithi007 bash[30436]: cluster 2023-12-16T07:43:13.986086+0000 mgr.x (mgr.24968) 24 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:16.435 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:43:16.435 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (6m) 7s ago 6m 19.3M - 0.23.0 ba2b418f427c 128c156c0190 2023-12-16T07:43:16.435 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (6m) 7s ago 6m 47.3M - 8.3.5 dad864ee21e9 25b9cf47977e 2023-12-16T07:43:16.435 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (4m) 7s ago 4m 47.7M - 3.5 e1d6a67b021e ca15814d070f 2023-12-16T07:43:16.435 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283 running (35s) 7s ago 12m 475M - 19.0.0-191-g6d933094 a29c258b5d6f e5055e9fae4f 2023-12-16T07:43:16.435 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:9283 running (14m) 7s ago 14m 404M - 17.2.0 e1d6a67b021e 5219d4698403 2023-12-16T07:43:16.435 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (15m) 7s ago 15m 62.2M 2048M 17.2.0 e1d6a67b021e c0468ecbd44e 2023-12-16T07:43:16.436 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (13m) 7s ago 13m 49.3M 2048M 17.2.0 e1d6a67b021e b47a960c6102 2023-12-16T07:43:16.436 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (13m) 7s ago 13m 45.6M 2048M 17.2.0 e1d6a67b021e c659b986f201 2023-12-16T07:43:16.436 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (7m) 7s ago 7m 12.5M - 1.3.1 1dbe0e931976 54a71535c2b4 2023-12-16T07:43:16.436 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (7m) 7s ago 7m 11.9M - 1.3.1 1dbe0e931976 b050faea525d 2023-12-16T07:43:16.436 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (12m) 7s ago 12m 56.6M 2782M 17.2.0 e1d6a67b021e dbc4d9538238 2023-12-16T07:43:16.436 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (11m) 7s ago 11m 58.0M 2782M 17.2.0 e1d6a67b021e 14637e0f389c 2023-12-16T07:43:16.436 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (11m) 7s ago 11m 55.6M 2782M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:43:16.436 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (10m) 7s ago 10m 54.8M 2782M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:43:16.436 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (9m) 7s ago 9m 56.1M 4317M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:43:16.436 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (9m) 7s ago 9m 54.1M 4317M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:43:16.436 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (8m) 7s ago 8m 53.8M 4317M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:43:16.437 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (8m) 7s ago 8m 53.9M 4317M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:43:16.437 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (6m) 7s ago 7m 60.2M - 2.33.4 514e6a882f6e 9351c4c7d0ba 2023-12-16T07:43:16.437 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (4m) 7s ago 4m 85.4M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:43:16.437 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (4m) 7s ago 4m 85.5M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:43:16.437 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (4m) 7s ago 4m 85.2M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:43:16.437 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (4m) 7s ago 4m 85.3M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:43:16.878 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:43:16.878 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:43:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2023-12-16T07:43:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:43:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:43:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2023-12-16T07:43:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 1 2023-12-16T07:43:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:43:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:43:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2023-12-16T07:43:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:43:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2023-12-16T07:43:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:43:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:43:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:43:16.879 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:43:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2023-12-16T07:43:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 1 2023-12-16T07:43:16.880 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:43:16.880 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:43:16.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:16 smithi169 bash[25083]: audit 2023-12-16T07:43:15.627994+0000 mgr.x (mgr.24968) 25 : audit [DBG] from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:17.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:16 smithi007 bash[30436]: audit 2023-12-16T07:43:15.627994+0000 mgr.x (mgr.24968) 25 : audit [DBG] from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:17.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:16 smithi007 bash[25009]: audit 2023-12-16T07:43:15.627994+0000 mgr.x (mgr.24968) 25 : audit [DBG] from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:17.265 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:43:17.265 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:43:17.265 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:43:17.265 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:43:17.265 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [], 2023-12-16T07:43:17.265 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "1/23 daemons upgraded", 2023-12-16T07:43:17.265 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "", 2023-12-16T07:43:17.265 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:43:17.266 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:43:17.710 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:43:18.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:17 smithi007 bash[25009]: cluster 2023-12-16T07:43:15.988132+0000 mgr.x (mgr.24968) 26 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:18.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:17 smithi007 bash[25009]: audit 2023-12-16T07:43:16.026807+0000 mgr.x (mgr.24968) 27 : audit [DBG] from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:18.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:17 smithi007 bash[25009]: audit 2023-12-16T07:43:16.403429+0000 mgr.x (mgr.24968) 28 : audit [DBG] from='client.25174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:18.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:17 smithi007 bash[25009]: audit 2023-12-16T07:43:16.876206+0000 mon.c (mon.1) 37 : audit [DBG] from='client.? 172.21.15.7:0/3642774470' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:43:18.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:17 smithi007 bash[25009]: audit 2023-12-16T07:43:17.263433+0000 mgr.x (mgr.24968) 29 : audit [DBG] from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:18.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:17 smithi007 bash[25009]: audit 2023-12-16T07:43:17.708669+0000 mon.a (mon.0) 936 : audit [DBG] from='client.? 172.21.15.7:0/1807695616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:43:18.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:17 smithi007 bash[30436]: cluster 2023-12-16T07:43:15.988132+0000 mgr.x (mgr.24968) 26 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:18.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:17 smithi007 bash[30436]: audit 2023-12-16T07:43:16.026807+0000 mgr.x (mgr.24968) 27 : audit [DBG] from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:18.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:17 smithi007 bash[30436]: audit 2023-12-16T07:43:16.403429+0000 mgr.x (mgr.24968) 28 : audit [DBG] from='client.25174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:18.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:17 smithi007 bash[30436]: audit 2023-12-16T07:43:16.876206+0000 mon.c (mon.1) 37 : audit [DBG] from='client.? 172.21.15.7:0/3642774470' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:43:18.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:17 smithi007 bash[30436]: audit 2023-12-16T07:43:17.263433+0000 mgr.x (mgr.24968) 29 : audit [DBG] from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:18.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:17 smithi007 bash[30436]: audit 2023-12-16T07:43:17.708669+0000 mon.a (mon.0) 936 : audit [DBG] from='client.? 172.21.15.7:0/1807695616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:43:18.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:17 smithi169 bash[25083]: cluster 2023-12-16T07:43:15.988132+0000 mgr.x (mgr.24968) 26 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:18.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:17 smithi169 bash[25083]: audit 2023-12-16T07:43:16.026807+0000 mgr.x (mgr.24968) 27 : audit [DBG] from='client.25168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:18.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:17 smithi169 bash[25083]: audit 2023-12-16T07:43:16.403429+0000 mgr.x (mgr.24968) 28 : audit [DBG] from='client.25174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:18.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:17 smithi169 bash[25083]: audit 2023-12-16T07:43:16.876206+0000 mon.c (mon.1) 37 : audit [DBG] from='client.? 172.21.15.7:0/3642774470' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:43:18.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:17 smithi169 bash[25083]: audit 2023-12-16T07:43:17.263433+0000 mgr.x (mgr.24968) 29 : audit [DBG] from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:18.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:17 smithi169 bash[25083]: audit 2023-12-16T07:43:17.708669+0000 mon.a (mon.0) 936 : audit [DBG] from='client.? 172.21.15.7:0/1807695616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:43:19.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:18 smithi007 bash[25009]: cluster 2023-12-16T07:43:17.988892+0000 mgr.x (mgr.24968) 30 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:19.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:18 smithi007 bash[30436]: cluster 2023-12-16T07:43:17.988892+0000 mgr.x (mgr.24968) 30 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:19.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:18 smithi169 bash[25083]: cluster 2023-12-16T07:43:17.988892+0000 mgr.x (mgr.24968) 30 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:20.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:19.519453+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:19.533672+0000 mon.a (mon.0) 938 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:19.538086+0000 mon.a (mon.0) 939 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:19.538598+0000 mon.b (mon.2) 54 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:19.540884+0000 mon.a (mon.0) 940 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:19.541538+0000 mon.b (mon.2) 55 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:19.543479+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:19.544192+0000 mon.b (mon.2) 56 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:19.545966+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:19.546851+0000 mon.b (mon.2) 57 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: cephadm 2023-12-16T07:43:19.548979+0000 mgr.x (mgr.24968) 31 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 3037M 2023-12-16T07:43:20.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:19.556802+0000 mon.a (mon.0) 943 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:19.560330+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:43:20.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:19.561985+0000 mon.b (mon.2) 59 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:43:20.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: cephadm 2023-12-16T07:43:19.564480+0000 mgr.x (mgr.24968) 32 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:43:20.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: cephadm 2023-12-16T07:43:19.564948+0000 mgr.x (mgr.24968) 33 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:43:20.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:20.234412+0000 mon.a (mon.0) 944 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:20.245536+0000 mon.a (mon.0) 945 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.823 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:20.283872+0000 mon.a (mon.0) 946 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.823 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:20.298476+0000 mon.a (mon.0) 947 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.823 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:20.310779+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:19.519453+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:19.533672+0000 mon.a (mon.0) 938 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:19.538086+0000 mon.a (mon.0) 939 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:19.538598+0000 mon.b (mon.2) 54 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:19.540884+0000 mon.a (mon.0) 940 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:19.541538+0000 mon.b (mon.2) 55 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:19.543479+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:19.544192+0000 mon.b (mon.2) 56 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:19.545966+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:19.546851+0000 mon.b (mon.2) 57 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: cephadm 2023-12-16T07:43:19.548979+0000 mgr.x (mgr.24968) 31 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 3037M 2023-12-16T07:43:20.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:19.556802+0000 mon.a (mon.0) 943 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:19.560330+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:43:20.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:19.561985+0000 mon.b (mon.2) 59 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:43:20.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: cephadm 2023-12-16T07:43:19.564480+0000 mgr.x (mgr.24968) 32 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:43:20.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: cephadm 2023-12-16T07:43:19.564948+0000 mgr.x (mgr.24968) 33 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:43:20.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:20.234412+0000 mon.a (mon.0) 944 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:20.245536+0000 mon.a (mon.0) 945 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:20.283872+0000 mon.a (mon.0) 946 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:20.298476+0000 mon.a (mon.0) 947 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:20.310779+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:20.410050+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:20.418706+0000 mon.a (mon.0) 950 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:20.425526+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[25009]: audit 2023-12-16T07:43:20.432911+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:20.410050+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:20.418706+0000 mon.a (mon.0) 950 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:20.425526+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[30436]: audit 2023-12-16T07:43:20.432911+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:19.519453+0000 mon.a (mon.0) 937 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:19.533672+0000 mon.a (mon.0) 938 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:19.538086+0000 mon.a (mon.0) 939 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:19.538598+0000 mon.b (mon.2) 54 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:19.540884+0000 mon.a (mon.0) 940 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:19.541538+0000 mon.b (mon.2) 55 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:19.543479+0000 mon.a (mon.0) 941 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:19.544192+0000 mon.b (mon.2) 56 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:19.545966+0000 mon.a (mon.0) 942 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:19.546851+0000 mon.b (mon.2) 57 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-12-16T07:43:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: cephadm 2023-12-16T07:43:19.548979+0000 mgr.x (mgr.24968) 31 : cephadm [INF] Adjusting osd_memory_target on smithi169 to 3037M 2023-12-16T07:43:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:19.556802+0000 mon.a (mon.0) 943 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:19.560330+0000 mon.b (mon.2) 58 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:43:20.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:19.561985+0000 mon.b (mon.2) 59 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:43:20.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: cephadm 2023-12-16T07:43:19.564480+0000 mgr.x (mgr.24968) 32 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:43:20.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: cephadm 2023-12-16T07:43:19.564948+0000 mgr.x (mgr.24968) 33 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:43:20.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:20.234412+0000 mon.a (mon.0) 944 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:20.245536+0000 mon.a (mon.0) 945 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:20.283872+0000 mon.a (mon.0) 946 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:20.298476+0000 mon.a (mon.0) 947 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:20.310779+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:20.410050+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:20.418706+0000 mon.a (mon.0) 950 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:20.425526+0000 mon.a (mon.0) 951 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:20.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:20 smithi169 bash[25083]: audit 2023-12-16T07:43:20.432911+0000 mon.a (mon.0) 952 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:21.246 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:20 smithi007 systemd[1]: Stopping Ceph node-exporter.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:43:21.246 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:20 smithi007 bash[59510]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-node-exporter.a 2023-12-16T07:43:21.817 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[59519]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-node-exporter-a 2023-12-16T07:43:21.817 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[59566]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-node-exporter.a 2023-12-16T07:43:21.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[25009]: cephadm 2023-12-16T07:43:19.730305+0000 mgr.x (mgr.24968) 34 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:43:21.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[25009]: cephadm 2023-12-16T07:43:19.780413+0000 mgr.x (mgr.24968) 35 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:43:21.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[25009]: cephadm 2023-12-16T07:43:19.895896+0000 mgr.x (mgr.24968) 36 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:43:21.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[25009]: cephadm 2023-12-16T07:43:19.945198+0000 mgr.x (mgr.24968) 37 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:43:21.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[25009]: cluster 2023-12-16T07:43:19.989713+0000 mgr.x (mgr.24968) 38 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:21.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[25009]: cephadm 2023-12-16T07:43:20.060630+0000 mgr.x (mgr.24968) 39 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:43:21.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[25009]: cephadm 2023-12-16T07:43:20.110439+0000 mgr.x (mgr.24968) 40 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:43:21.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[25009]: cephadm 2023-12-16T07:43:20.438623+0000 mgr.x (mgr.24968) 41 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2023-12-16T07:43:21.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[25009]: cephadm 2023-12-16T07:43:20.439718+0000 mgr.x (mgr.24968) 42 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi007 2023-12-16T07:43:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[30436]: cephadm 2023-12-16T07:43:19.730305+0000 mgr.x (mgr.24968) 34 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:43:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[30436]: cephadm 2023-12-16T07:43:19.780413+0000 mgr.x (mgr.24968) 35 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:43:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[30436]: cephadm 2023-12-16T07:43:19.895896+0000 mgr.x (mgr.24968) 36 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:43:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[30436]: cephadm 2023-12-16T07:43:19.945198+0000 mgr.x (mgr.24968) 37 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:43:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[30436]: cluster 2023-12-16T07:43:19.989713+0000 mgr.x (mgr.24968) 38 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:21.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[30436]: cephadm 2023-12-16T07:43:20.060630+0000 mgr.x (mgr.24968) 39 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:43:21.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[30436]: cephadm 2023-12-16T07:43:20.110439+0000 mgr.x (mgr.24968) 40 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:43:21.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[30436]: cephadm 2023-12-16T07:43:20.438623+0000 mgr.x (mgr.24968) 41 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2023-12-16T07:43:21.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:21 smithi007 bash[30436]: cephadm 2023-12-16T07:43:20.439718+0000 mgr.x (mgr.24968) 42 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi007 2023-12-16T07:43:21.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:21 smithi169 bash[25083]: cephadm 2023-12-16T07:43:19.730305+0000 mgr.x (mgr.24968) 34 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:43:21.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:21 smithi169 bash[25083]: cephadm 2023-12-16T07:43:19.780413+0000 mgr.x (mgr.24968) 35 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:43:21.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:21 smithi169 bash[25083]: cephadm 2023-12-16T07:43:19.895896+0000 mgr.x (mgr.24968) 36 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:43:21.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:21 smithi169 bash[25083]: cephadm 2023-12-16T07:43:19.945198+0000 mgr.x (mgr.24968) 37 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:43:21.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:21 smithi169 bash[25083]: cluster 2023-12-16T07:43:19.989713+0000 mgr.x (mgr.24968) 38 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:21.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:21 smithi169 bash[25083]: cephadm 2023-12-16T07:43:20.060630+0000 mgr.x (mgr.24968) 39 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:43:21.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:21 smithi169 bash[25083]: cephadm 2023-12-16T07:43:20.110439+0000 mgr.x (mgr.24968) 40 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:43:21.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:21 smithi169 bash[25083]: cephadm 2023-12-16T07:43:20.438623+0000 mgr.x (mgr.24968) 41 : cephadm [INF] Reconfiguring node-exporter.a (dependencies changed)... 2023-12-16T07:43:21.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:21 smithi169 bash[25083]: cephadm 2023-12-16T07:43:20.439718+0000 mgr.x (mgr.24968) 42 : cephadm [INF] Reconfiguring daemon node-exporter.a on smithi007 2023-12-16T07:43:22.317 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:21 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2023-12-16T07:43:22.317 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:21 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2023-12-16T07:43:22.317 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:21 smithi007 systemd[1]: Stopped Ceph node-exporter.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:43:22.317 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:21 smithi007 systemd[1]: Started Ceph node-exporter.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:43:22.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:43:22 smithi169 bash[45188]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:43:22] "GET /metrics HTTP/1.1" 200 38136 "" "Prometheus/2.33.4" 2023-12-16T07:43:22.683 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:22 smithi169 bash[41600]: ts=2023-12-16T07:43:22.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.7:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.169:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:43:23.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:22 smithi007 bash[30436]: cluster 2023-12-16T07:43:21.991437+0000 mgr.x (mgr.24968) 43 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:23.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:22 smithi007 bash[30436]: audit 2023-12-16T07:43:21.996455+0000 mon.a (mon.0) 953 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:23.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:22 smithi007 bash[30436]: audit 2023-12-16T07:43:22.008062+0000 mon.a (mon.0) 954 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:23.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:22 smithi007 bash[30436]: cephadm 2023-12-16T07:43:22.059679+0000 mgr.x (mgr.24968) 44 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-12-16T07:43:23.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:22 smithi007 bash[30436]: cephadm 2023-12-16T07:43:22.081356+0000 mgr.x (mgr.24968) 45 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi007 2023-12-16T07:43:23.345 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.168Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2023-12-16T07:43:23.346 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.168Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2023-12-16T07:43:23.346 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.168Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2023-12-16T07:43:23.346 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.168Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2023-12-16T07:43:23.346 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.169Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2023-12-16T07:43:23.346 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.169Z caller=node_exporter.go:115 level=info collector=arp 2023-12-16T07:43:23.346 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.169Z caller=node_exporter.go:115 level=info collector=bcache 2023-12-16T07:43:23.346 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.169Z caller=node_exporter.go:115 level=info collector=bonding 2023-12-16T07:43:23.346 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.169Z caller=node_exporter.go:115 level=info collector=btrfs 2023-12-16T07:43:23.346 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.169Z caller=node_exporter.go:115 level=info collector=conntrack 2023-12-16T07:43:23.346 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.169Z caller=node_exporter.go:115 level=info collector=cpu 2023-12-16T07:43:23.346 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.169Z caller=node_exporter.go:115 level=info collector=cpufreq 2023-12-16T07:43:23.346 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.169Z caller=node_exporter.go:115 level=info collector=diskstats 2023-12-16T07:43:23.347 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.169Z caller=node_exporter.go:115 level=info collector=dmi 2023-12-16T07:43:23.347 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.169Z caller=node_exporter.go:115 level=info collector=edac 2023-12-16T07:43:23.347 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.169Z caller=node_exporter.go:115 level=info collector=entropy 2023-12-16T07:43:23.347 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=fibrechannel 2023-12-16T07:43:23.347 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=filefd 2023-12-16T07:43:23.347 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=filesystem 2023-12-16T07:43:23.347 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=hwmon 2023-12-16T07:43:23.347 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=infiniband 2023-12-16T07:43:23.347 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=ipvs 2023-12-16T07:43:23.347 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=loadavg 2023-12-16T07:43:23.347 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=mdadm 2023-12-16T07:43:23.347 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=meminfo 2023-12-16T07:43:23.347 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=netclass 2023-12-16T07:43:23.347 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=netdev 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=netstat 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=nfs 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=nfsd 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=nvme 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=os 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=pressure 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=rapl 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=schedstat 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=sockstat 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=softnet 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=stat 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=tapestats 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=textfile 2023-12-16T07:43:23.348 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=thermal_zone 2023-12-16T07:43:23.349 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=time 2023-12-16T07:43:23.349 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=udp_queues 2023-12-16T07:43:23.349 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=uname 2023-12-16T07:43:23.349 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=vmstat 2023-12-16T07:43:23.349 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=xfs 2023-12-16T07:43:23.349 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:115 level=info collector=zfs 2023-12-16T07:43:23.349 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2023-12-16T07:43:23.349 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[59594]: ts=2023-12-16T07:43:23.170Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2023-12-16T07:43:23.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:22 smithi007 bash[25009]: cluster 2023-12-16T07:43:21.991437+0000 mgr.x (mgr.24968) 43 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:23.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:22 smithi007 bash[25009]: audit 2023-12-16T07:43:21.996455+0000 mon.a (mon.0) 953 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:23.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:22 smithi007 bash[25009]: audit 2023-12-16T07:43:22.008062+0000 mon.a (mon.0) 954 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:23.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:22 smithi007 bash[25009]: cephadm 2023-12-16T07:43:22.059679+0000 mgr.x (mgr.24968) 44 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-12-16T07:43:23.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:22 smithi007 bash[25009]: cephadm 2023-12-16T07:43:22.081356+0000 mgr.x (mgr.24968) 45 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi007 2023-12-16T07:43:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:22 smithi169 bash[25083]: cluster 2023-12-16T07:43:21.991437+0000 mgr.x (mgr.24968) 43 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:22 smithi169 bash[25083]: audit 2023-12-16T07:43:21.996455+0000 mon.a (mon.0) 953 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:22 smithi169 bash[25083]: audit 2023-12-16T07:43:22.008062+0000 mon.a (mon.0) 954 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:22 smithi169 bash[25083]: cephadm 2023-12-16T07:43:22.059679+0000 mgr.x (mgr.24968) 44 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2023-12-16T07:43:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:22 smithi169 bash[25083]: cephadm 2023-12-16T07:43:22.081356+0000 mgr.x (mgr.24968) 45 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi007 2023-12-16T07:43:23.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:43:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:43:23.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[49264]: level=error ts=2023-12-16T07:43:23.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:43:23.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[49264]: level=warn ts=2023-12-16T07:43:23.524Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2023-12-16T07:43:23.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:23 smithi007 bash[49264]: level=warn ts=2023-12-16T07:43:23.525Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.169:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.169 because it doesn't contain any IP SANs" 2023-12-16T07:43:25.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:24 smithi007 bash[25009]: audit 2023-12-16T07:43:23.816800+0000 mgr.x (mgr.24968) 46 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:25.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:24 smithi007 bash[25009]: cluster 2023-12-16T07:43:23.992181+0000 mgr.x (mgr.24968) 47 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:25.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:24 smithi007 bash[25009]: audit 2023-12-16T07:43:24.164320+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:43:25.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:24 smithi007 bash[30436]: audit 2023-12-16T07:43:23.816800+0000 mgr.x (mgr.24968) 46 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:25.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:24 smithi007 bash[30436]: cluster 2023-12-16T07:43:23.992181+0000 mgr.x (mgr.24968) 47 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:25.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:24 smithi007 bash[30436]: audit 2023-12-16T07:43:24.164320+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:43:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:25 smithi169 bash[25083]: audit 2023-12-16T07:43:23.816800+0000 mgr.x (mgr.24968) 46 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:25 smithi169 bash[25083]: cluster 2023-12-16T07:43:23.992181+0000 mgr.x (mgr.24968) 47 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:25 smithi169 bash[25083]: audit 2023-12-16T07:43:24.164320+0000 mon.b (mon.2) 60 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:43:27.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:27 smithi007 bash[25009]: cluster 2023-12-16T07:43:25.993914+0000 mgr.x (mgr.24968) 48 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:27.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:27 smithi007 bash[30436]: cluster 2023-12-16T07:43:25.993914+0000 mgr.x (mgr.24968) 48 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:27 smithi169 bash[25083]: cluster 2023-12-16T07:43:25.993914+0000 mgr.x (mgr.24968) 48 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:29.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:29 smithi007 bash[25009]: cluster 2023-12-16T07:43:27.994732+0000 mgr.x (mgr.24968) 49 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:29.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:29 smithi007 bash[30436]: cluster 2023-12-16T07:43:27.994732+0000 mgr.x (mgr.24968) 49 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:29.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:29 smithi169 bash[25083]: cluster 2023-12-16T07:43:27.994732+0000 mgr.x (mgr.24968) 49 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:31.514 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:31 smithi007 bash[30436]: cluster 2023-12-16T07:43:29.995565+0000 mgr.x (mgr.24968) 50 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:31.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:31 smithi007 bash[25009]: cluster 2023-12-16T07:43:29.995565+0000 mgr.x (mgr.24968) 50 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:31.514 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:31 smithi007 systemd[1]: Stopping Ceph alertmanager.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:43:31.514 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:31 smithi007 bash[59875]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-alertmanager.a 2023-12-16T07:43:31.515 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:31 smithi007 bash[49264]: level=info ts=2023-12-16T07:43:31.282Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2023-12-16T07:43:31.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:31 smithi169 bash[25083]: cluster 2023-12-16T07:43:29.995565+0000 mgr.x (mgr.24968) 50 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:32.160 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:31 smithi007 bash[59885]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-alertmanager-a 2023-12-16T07:43:32.161 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:31 smithi007 bash[59921]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-alertmanager.a 2023-12-16T07:43:32.567 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:32 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@alertmanager.a.service: Succeeded. 2023-12-16T07:43:32.567 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:32 smithi007 systemd[1]: Stopped Ceph alertmanager.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:43:32.567 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:32 smithi007 systemd[1]: Started Ceph alertmanager.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:43:32.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:43:32 smithi169 bash[45188]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:43:32] "GET /metrics HTTP/1.1" 200 38136 "" "Prometheus/2.33.4" 2023-12-16T07:43:32.683 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:32 smithi169 bash[41600]: ts=2023-12-16T07:43:32.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.7:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.169:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:43:33.514 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:33 smithi169 bash[25083]: cluster 2023-12-16T07:43:31.997175+0000 mgr.x (mgr.24968) 51 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:33.515 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:33 smithi169 bash[25083]: audit 2023-12-16T07:43:32.214322+0000 mon.a (mon.0) 955 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:33.515 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:33 smithi169 bash[25083]: audit 2023-12-16T07:43:32.223378+0000 mon.a (mon.0) 956 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:33.515 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:33 smithi169 bash[25083]: cephadm 2023-12-16T07:43:32.272856+0000 mgr.x (mgr.24968) 52 : cephadm [INF] Reconfiguring iscsi.foo.smithi007.crrdaq (dependencies changed)... 2023-12-16T07:43:33.515 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:33 smithi169 bash[25083]: audit 2023-12-16T07:43:32.273134+0000 mon.a (mon.0) 957 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-12-16T07:43:33.515 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:33 smithi169 bash[25083]: audit 2023-12-16T07:43:32.273719+0000 mon.b (mon.2) 61 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-12-16T07:43:33.515 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:33 smithi169 bash[25083]: audit 2023-12-16T07:43:32.288262+0000 mon.b (mon.2) 62 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:43:33.515 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:33 smithi169 bash[25083]: cephadm 2023-12-16T07:43:32.290366+0000 mgr.x (mgr.24968) 53 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi007.crrdaq on smithi007 2023-12-16T07:43:33.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[25009]: cluster 2023-12-16T07:43:31.997175+0000 mgr.x (mgr.24968) 51 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:33.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[25009]: audit 2023-12-16T07:43:32.214322+0000 mon.a (mon.0) 955 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:33.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[25009]: audit 2023-12-16T07:43:32.223378+0000 mon.a (mon.0) 956 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:33.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[25009]: cephadm 2023-12-16T07:43:32.272856+0000 mgr.x (mgr.24968) 52 : cephadm [INF] Reconfiguring iscsi.foo.smithi007.crrdaq (dependencies changed)... 2023-12-16T07:43:33.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[25009]: audit 2023-12-16T07:43:32.273134+0000 mon.a (mon.0) 957 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-12-16T07:43:33.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[25009]: audit 2023-12-16T07:43:32.273719+0000 mon.b (mon.2) 61 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-12-16T07:43:33.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[25009]: audit 2023-12-16T07:43:32.288262+0000 mon.b (mon.2) 62 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:43:33.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[25009]: cephadm 2023-12-16T07:43:32.290366+0000 mgr.x (mgr.24968) 53 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi007.crrdaq on smithi007 2023-12-16T07:43:33.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[30436]: cluster 2023-12-16T07:43:31.997175+0000 mgr.x (mgr.24968) 51 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:33.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[30436]: audit 2023-12-16T07:43:32.214322+0000 mon.a (mon.0) 955 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:33.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[30436]: audit 2023-12-16T07:43:32.223378+0000 mon.a (mon.0) 956 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:33.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[30436]: cephadm 2023-12-16T07:43:32.272856+0000 mgr.x (mgr.24968) 52 : cephadm [INF] Reconfiguring iscsi.foo.smithi007.crrdaq (dependencies changed)... 2023-12-16T07:43:33.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[30436]: audit 2023-12-16T07:43:32.273134+0000 mon.a (mon.0) 957 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-12-16T07:43:33.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[30436]: audit 2023-12-16T07:43:32.273719+0000 mon.b (mon.2) 61 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-12-16T07:43:33.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[30436]: audit 2023-12-16T07:43:32.288262+0000 mon.b (mon.2) 62 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:43:33.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[30436]: cephadm 2023-12-16T07:43:32.290366+0000 mgr.x (mgr.24968) 53 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi007.crrdaq on smithi007 2023-12-16T07:43:33.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:33 smithi169 bash[41600]: ts=2023-12-16T07:43:33.514Z caller=notifier.go:526 level=error component=notifier alertmanager=http://172.21.15.7:9093/api/v2/alerts count=6 msg="Error sending alert" err="Post \"http://172.21.15.7:9093/api/v2/alerts\": dial tcp 172.21.15.7:9093: connect: connection refused" 2023-12-16T07:43:34.085 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:43:33 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:43:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:43:34.720 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:34 smithi007 bash[59950]: level=info ts=2023-12-16T07:43:34.567Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2023-12-16T07:43:34.720 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:34 smithi007 bash[59950]: level=info ts=2023-12-16T07:43:34.567Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2023-12-16T07:43:34.720 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:34 smithi007 bash[59950]: level=info ts=2023-12-16T07:43:34.569Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.7 port=9094 2023-12-16T07:43:34.720 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:34 smithi007 bash[59950]: level=info ts=2023-12-16T07:43:34.572Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2023-12-16T07:43:34.720 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:34 smithi007 bash[59950]: level=info ts=2023-12-16T07:43:34.620Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2023-12-16T07:43:34.720 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:34 smithi007 bash[59950]: level=info ts=2023-12-16T07:43:34.620Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2023-12-16T07:43:34.720 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:34 smithi007 bash[59950]: level=info ts=2023-12-16T07:43:34.625Z caller=main.go:518 msg=Listening address=:9093 2023-12-16T07:43:34.720 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:34 smithi007 bash[59950]: level=info ts=2023-12-16T07:43:34.626Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2023-12-16T07:43:35.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:35 smithi007 bash[25009]: audit 2023-12-16T07:43:33.818365+0000 mgr.x (mgr.24968) 54 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:35.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:35 smithi007 bash[25009]: cluster 2023-12-16T07:43:33.997926+0000 mgr.x (mgr.24968) 55 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:35.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:35 smithi007 bash[30436]: audit 2023-12-16T07:43:33.818365+0000 mgr.x (mgr.24968) 54 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:35.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:35 smithi007 bash[30436]: cluster 2023-12-16T07:43:33.997926+0000 mgr.x (mgr.24968) 55 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:35 smithi169 bash[25083]: audit 2023-12-16T07:43:33.818365+0000 mgr.x (mgr.24968) 54 : audit [DBG] from='client.24788 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:35 smithi169 bash[25083]: cluster 2023-12-16T07:43:33.997926+0000 mgr.x (mgr.24968) 55 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:36.614 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:36 smithi007 bash[59950]: level=info ts=2023-12-16T07:43:36.574Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000985084s 2023-12-16T07:43:37.440 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:37 smithi007 bash[30436]: cluster 2023-12-16T07:43:35.999677+0000 mgr.x (mgr.24968) 56 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:37.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:37 smithi007 bash[25009]: cluster 2023-12-16T07:43:35.999677+0000 mgr.x (mgr.24968) 56 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:37.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:37 smithi169 bash[25083]: cluster 2023-12-16T07:43:35.999677+0000 mgr.x (mgr.24968) 56 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:38 smithi169 bash[25083]: audit 2023-12-16T07:43:37.370268+0000 mon.a (mon.0) 958 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:38 smithi169 bash[25083]: audit 2023-12-16T07:43:37.381804+0000 mon.a (mon.0) 959 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:38 smithi169 bash[25083]: cephadm 2023-12-16T07:43:37.419119+0000 mgr.x (mgr.24968) 57 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-12-16T07:43:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:38 smithi169 bash[25083]: cephadm 2023-12-16T07:43:37.660399+0000 mgr.x (mgr.24968) 58 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi169 2023-12-16T07:43:38.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:38 smithi007 bash[25009]: audit 2023-12-16T07:43:37.370268+0000 mon.a (mon.0) 958 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:38.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:38 smithi007 bash[25009]: audit 2023-12-16T07:43:37.381804+0000 mon.a (mon.0) 959 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:38.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:38 smithi007 bash[25009]: cephadm 2023-12-16T07:43:37.419119+0000 mgr.x (mgr.24968) 57 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-12-16T07:43:38.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:38 smithi007 bash[25009]: cephadm 2023-12-16T07:43:37.660399+0000 mgr.x (mgr.24968) 58 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi169 2023-12-16T07:43:38.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:38 smithi007 bash[30436]: audit 2023-12-16T07:43:37.370268+0000 mon.a (mon.0) 958 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:38.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:38 smithi007 bash[30436]: audit 2023-12-16T07:43:37.381804+0000 mon.a (mon.0) 959 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:38.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:38 smithi007 bash[30436]: cephadm 2023-12-16T07:43:37.419119+0000 mgr.x (mgr.24968) 57 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-12-16T07:43:38.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:38 smithi007 bash[30436]: cephadm 2023-12-16T07:43:37.660399+0000 mgr.x (mgr.24968) 58 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi169 2023-12-16T07:43:39.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:39 smithi007 bash[25009]: cluster 2023-12-16T07:43:38.000420+0000 mgr.x (mgr.24968) 59 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:39.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:39 smithi007 bash[25009]: audit 2023-12-16T07:43:39.164629+0000 mon.b (mon.2) 63 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:43:39.500 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:39 smithi007 bash[30436]: cluster 2023-12-16T07:43:38.000420+0000 mgr.x (mgr.24968) 59 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:39.500 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:39 smithi007 bash[30436]: audit 2023-12-16T07:43:39.164629+0000 mon.b (mon.2) 63 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:43:39.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:39 smithi169 bash[25083]: cluster 2023-12-16T07:43:38.000420+0000 mgr.x (mgr.24968) 59 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:39.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:39 smithi169 bash[25083]: audit 2023-12-16T07:43:39.164629+0000 mon.b (mon.2) 63 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:43:40.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:40 smithi169 bash[25083]: audit 2023-12-16T07:43:39.863449+0000 mon.a (mon.0) 960 : audit [DBG] from='client.? 172.21.15.7:0/760632020' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-12-16T07:43:40.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:40 smithi169 bash[25083]: audit 2023-12-16T07:43:40.256576+0000 mon.c (mon.1) 38 : audit [INF] from='client.? 172.21.15.7:0/3010909415' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2952790388"}]: dispatch 2023-12-16T07:43:40.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:40 smithi169 bash[25083]: audit 2023-12-16T07:43:40.257548+0000 mon.a (mon.0) 961 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2952790388"}]: dispatch 2023-12-16T07:43:40.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:40 smithi007 bash[25009]: audit 2023-12-16T07:43:39.863449+0000 mon.a (mon.0) 960 : audit [DBG] from='client.? 172.21.15.7:0/760632020' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-12-16T07:43:40.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:40 smithi007 bash[25009]: audit 2023-12-16T07:43:40.256576+0000 mon.c (mon.1) 38 : audit [INF] from='client.? 172.21.15.7:0/3010909415' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2952790388"}]: dispatch 2023-12-16T07:43:40.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:40 smithi007 bash[25009]: audit 2023-12-16T07:43:40.257548+0000 mon.a (mon.0) 961 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2952790388"}]: dispatch 2023-12-16T07:43:40.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:40 smithi007 bash[30436]: audit 2023-12-16T07:43:39.863449+0000 mon.a (mon.0) 960 : audit [DBG] from='client.? 172.21.15.7:0/760632020' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-12-16T07:43:40.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:40 smithi007 bash[30436]: audit 2023-12-16T07:43:40.256576+0000 mon.c (mon.1) 38 : audit [INF] from='client.? 172.21.15.7:0/3010909415' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2952790388"}]: dispatch 2023-12-16T07:43:40.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:40 smithi007 bash[30436]: audit 2023-12-16T07:43:40.257548+0000 mon.a (mon.0) 961 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2952790388"}]: dispatch 2023-12-16T07:43:41.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:41 smithi169 bash[25083]: cluster 2023-12-16T07:43:40.001113+0000 mgr.x (mgr.24968) 60 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:41.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:41 smithi169 bash[25083]: audit 2023-12-16T07:43:40.396791+0000 mon.a (mon.0) 962 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2952790388"}]': finished 2023-12-16T07:43:41.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:41 smithi169 bash[25083]: cluster 2023-12-16T07:43:40.396913+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2023-12-16T07:43:41.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:41 smithi169 bash[25083]: audit 2023-12-16T07:43:40.794747+0000 mon.c (mon.1) 39 : audit [INF] from='client.? 172.21.15.7:0/194027428' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/906851426"}]: dispatch 2023-12-16T07:43:41.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:41 smithi169 bash[25083]: audit 2023-12-16T07:43:40.795459+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/906851426"}]: dispatch 2023-12-16T07:43:41.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:41 smithi007 bash[25009]: cluster 2023-12-16T07:43:40.001113+0000 mgr.x (mgr.24968) 60 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:41.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:41 smithi007 bash[25009]: audit 2023-12-16T07:43:40.396791+0000 mon.a (mon.0) 962 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2952790388"}]': finished 2023-12-16T07:43:41.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:41 smithi007 bash[25009]: cluster 2023-12-16T07:43:40.396913+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2023-12-16T07:43:41.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:41 smithi007 bash[25009]: audit 2023-12-16T07:43:40.794747+0000 mon.c (mon.1) 39 : audit [INF] from='client.? 172.21.15.7:0/194027428' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/906851426"}]: dispatch 2023-12-16T07:43:41.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:41 smithi007 bash[25009]: audit 2023-12-16T07:43:40.795459+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/906851426"}]: dispatch 2023-12-16T07:43:41.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:41 smithi007 bash[30436]: cluster 2023-12-16T07:43:40.001113+0000 mgr.x (mgr.24968) 60 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:43:41.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:41 smithi007 bash[30436]: audit 2023-12-16T07:43:40.396791+0000 mon.a (mon.0) 962 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2952790388"}]': finished 2023-12-16T07:43:41.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:41 smithi007 bash[30436]: cluster 2023-12-16T07:43:40.396913+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2023-12-16T07:43:41.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:41 smithi007 bash[30436]: audit 2023-12-16T07:43:40.794747+0000 mon.c (mon.1) 39 : audit [INF] from='client.? 172.21.15.7:0/194027428' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/906851426"}]: dispatch 2023-12-16T07:43:41.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:41 smithi007 bash[30436]: audit 2023-12-16T07:43:40.795459+0000 mon.a (mon.0) 964 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/906851426"}]: dispatch 2023-12-16T07:43:42.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:43:42 smithi169 bash[45188]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:43:42] "GET /metrics HTTP/1.1" 200 38130 "" "Prometheus/2.33.4" 2023-12-16T07:43:42.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:42 smithi169 bash[25083]: audit 2023-12-16T07:43:41.405830+0000 mon.a (mon.0) 965 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/906851426"}]': finished 2023-12-16T07:43:42.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:42 smithi169 bash[25083]: cluster 2023-12-16T07:43:41.406012+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2023-12-16T07:43:42.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:42 smithi169 bash[25083]: audit 2023-12-16T07:43:41.802964+0000 mon.c (mon.1) 40 : audit [INF] from='client.? 172.21.15.7:0/2429826400' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2851654539"}]: dispatch 2023-12-16T07:43:42.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:42 smithi169 bash[25083]: audit 2023-12-16T07:43:41.803964+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2851654539"}]: dispatch 2023-12-16T07:43:42.684 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:42 smithi169 bash[41600]: ts=2023-12-16T07:43:42.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.7:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.169:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:43:42.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:42 smithi007 bash[25009]: audit 2023-12-16T07:43:41.405830+0000 mon.a (mon.0) 965 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/906851426"}]': finished 2023-12-16T07:43:42.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:42 smithi007 bash[25009]: cluster 2023-12-16T07:43:41.406012+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2023-12-16T07:43:42.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:42 smithi007 bash[25009]: audit 2023-12-16T07:43:41.802964+0000 mon.c (mon.1) 40 : audit [INF] from='client.? 172.21.15.7:0/2429826400' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2851654539"}]: dispatch 2023-12-16T07:43:42.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:42 smithi007 bash[25009]: audit 2023-12-16T07:43:41.803964+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2851654539"}]: dispatch 2023-12-16T07:43:42.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:42 smithi007 bash[30436]: audit 2023-12-16T07:43:41.405830+0000 mon.a (mon.0) 965 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/906851426"}]': finished 2023-12-16T07:43:42.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:42 smithi007 bash[30436]: cluster 2023-12-16T07:43:41.406012+0000 mon.a (mon.0) 966 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2023-12-16T07:43:42.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:42 smithi007 bash[30436]: audit 2023-12-16T07:43:41.802964+0000 mon.c (mon.1) 40 : audit [INF] from='client.? 172.21.15.7:0/2429826400' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2851654539"}]: dispatch 2023-12-16T07:43:42.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:42 smithi007 bash[30436]: audit 2023-12-16T07:43:41.803964+0000 mon.a (mon.0) 967 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2851654539"}]: dispatch 2023-12-16T07:43:43.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:43 smithi169 bash[25083]: cluster 2023-12-16T07:43:42.001832+0000 mgr.x (mgr.24968) 61 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-12-16T07:43:43.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:43 smithi169 bash[25083]: audit 2023-12-16T07:43:42.415540+0000 mon.a (mon.0) 968 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2851654539"}]': finished 2023-12-16T07:43:43.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:43 smithi169 bash[25083]: cluster 2023-12-16T07:43:42.415681+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2023-12-16T07:43:43.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:43 smithi169 bash[25083]: audit 2023-12-16T07:43:42.801942+0000 mon.c (mon.1) 41 : audit [INF] from='client.? 172.21.15.7:0/1813469399' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/942562873"}]: dispatch 2023-12-16T07:43:43.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:43 smithi169 bash[25083]: audit 2023-12-16T07:43:42.802923+0000 mon.a (mon.0) 970 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/942562873"}]: dispatch 2023-12-16T07:43:43.716 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:43 smithi007 bash[30436]: cluster 2023-12-16T07:43:42.001832+0000 mgr.x (mgr.24968) 61 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-12-16T07:43:43.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:43 smithi007 bash[30436]: audit 2023-12-16T07:43:42.415540+0000 mon.a (mon.0) 968 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2851654539"}]': finished 2023-12-16T07:43:43.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:43 smithi007 bash[30436]: cluster 2023-12-16T07:43:42.415681+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2023-12-16T07:43:43.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:43 smithi007 bash[30436]: audit 2023-12-16T07:43:42.801942+0000 mon.c (mon.1) 41 : audit [INF] from='client.? 172.21.15.7:0/1813469399' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/942562873"}]: dispatch 2023-12-16T07:43:43.717 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:43 smithi007 bash[30436]: audit 2023-12-16T07:43:42.802923+0000 mon.a (mon.0) 970 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/942562873"}]: dispatch 2023-12-16T07:43:43.717 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:43 smithi007 bash[25009]: cluster 2023-12-16T07:43:42.001832+0000 mgr.x (mgr.24968) 61 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-12-16T07:43:43.717 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:43 smithi007 bash[25009]: audit 2023-12-16T07:43:42.415540+0000 mon.a (mon.0) 968 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2851654539"}]': finished 2023-12-16T07:43:43.717 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:43 smithi007 bash[25009]: cluster 2023-12-16T07:43:42.415681+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2023-12-16T07:43:43.717 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:43 smithi007 bash[25009]: audit 2023-12-16T07:43:42.801942+0000 mon.c (mon.1) 41 : audit [INF] from='client.? 172.21.15.7:0/1813469399' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/942562873"}]: dispatch 2023-12-16T07:43:43.717 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:43 smithi007 bash[25009]: audit 2023-12-16T07:43:42.802923+0000 mon.a (mon.0) 970 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/942562873"}]: dispatch 2023-12-16T07:43:44.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:43:43 smithi007 bash[25282]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:43:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:43:44.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:44 smithi169 bash[25083]: audit 2023-12-16T07:43:43.419477+0000 mon.a (mon.0) 971 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/942562873"}]': finished 2023-12-16T07:43:44.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:44 smithi169 bash[25083]: cluster 2023-12-16T07:43:43.419628+0000 mon.a (mon.0) 972 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2023-12-16T07:43:44.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:44 smithi169 bash[25083]: audit 2023-12-16T07:43:43.813411+0000 mon.c (mon.1) 42 : audit [INF] from='client.? 172.21.15.7:0/498298135' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/942562873"}]: dispatch 2023-12-16T07:43:44.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:44 smithi169 bash[25083]: audit 2023-12-16T07:43:43.814336+0000 mon.a (mon.0) 973 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/942562873"}]: dispatch 2023-12-16T07:43:44.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:43:44 smithi007 bash[59950]: level=info ts=2023-12-16T07:43:44.576Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003514176s 2023-12-16T07:43:44.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:44 smithi007 bash[25009]: audit 2023-12-16T07:43:43.419477+0000 mon.a (mon.0) 971 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/942562873"}]': finished 2023-12-16T07:43:44.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:44 smithi007 bash[25009]: cluster 2023-12-16T07:43:43.419628+0000 mon.a (mon.0) 972 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2023-12-16T07:43:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:44 smithi007 bash[25009]: audit 2023-12-16T07:43:43.813411+0000 mon.c (mon.1) 42 : audit [INF] from='client.? 172.21.15.7:0/498298135' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/942562873"}]: dispatch 2023-12-16T07:43:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:44 smithi007 bash[25009]: audit 2023-12-16T07:43:43.814336+0000 mon.a (mon.0) 973 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/942562873"}]: dispatch 2023-12-16T07:43:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:44 smithi007 bash[30436]: audit 2023-12-16T07:43:43.419477+0000 mon.a (mon.0) 971 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/942562873"}]': finished 2023-12-16T07:43:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:44 smithi007 bash[30436]: cluster 2023-12-16T07:43:43.419628+0000 mon.a (mon.0) 972 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2023-12-16T07:43:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:44 smithi007 bash[30436]: audit 2023-12-16T07:43:43.813411+0000 mon.c (mon.1) 42 : audit [INF] from='client.? 172.21.15.7:0/498298135' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/942562873"}]: dispatch 2023-12-16T07:43:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:44 smithi007 bash[30436]: audit 2023-12-16T07:43:43.814336+0000 mon.a (mon.0) 973 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/942562873"}]: dispatch 2023-12-16T07:43:45.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:45 smithi007 bash[30436]: cluster 2023-12-16T07:43:44.002659+0000 mgr.x (mgr.24968) 62 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2023-12-16T07:43:45.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:45 smithi007 bash[30436]: audit 2023-12-16T07:43:44.437632+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/942562873"}]': finished 2023-12-16T07:43:45.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:45 smithi007 bash[30436]: cluster 2023-12-16T07:43:44.437849+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2023-12-16T07:43:45.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:45 smithi007 bash[30436]: audit 2023-12-16T07:43:44.825976+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.7:0/2161077206' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1908420919"}]: dispatch 2023-12-16T07:43:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:45 smithi007 bash[25009]: cluster 2023-12-16T07:43:44.002659+0000 mgr.x (mgr.24968) 62 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2023-12-16T07:43:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:45 smithi007 bash[25009]: audit 2023-12-16T07:43:44.437632+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/942562873"}]': finished 2023-12-16T07:43:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:45 smithi007 bash[25009]: cluster 2023-12-16T07:43:44.437849+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2023-12-16T07:43:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:45 smithi007 bash[25009]: audit 2023-12-16T07:43:44.825976+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.7:0/2161077206' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1908420919"}]: dispatch 2023-12-16T07:43:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:45 smithi169 bash[25083]: cluster 2023-12-16T07:43:44.002659+0000 mgr.x (mgr.24968) 62 : cluster [DBG] pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2023-12-16T07:43:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:45 smithi169 bash[25083]: audit 2023-12-16T07:43:44.437632+0000 mon.a (mon.0) 974 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/942562873"}]': finished 2023-12-16T07:43:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:45 smithi169 bash[25083]: cluster 2023-12-16T07:43:44.437849+0000 mon.a (mon.0) 975 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2023-12-16T07:43:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:45 smithi169 bash[25083]: audit 2023-12-16T07:43:44.825976+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.7:0/2161077206' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1908420919"}]: dispatch 2023-12-16T07:43:46.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:46 smithi007 bash[30436]: audit 2023-12-16T07:43:45.442932+0000 mon.a (mon.0) 977 : audit [INF] from='client.? 172.21.15.7:0/2161077206' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1908420919"}]': finished 2023-12-16T07:43:46.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:46 smithi007 bash[30436]: cluster 2023-12-16T07:43:45.443032+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2023-12-16T07:43:46.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:46 smithi007 bash[30436]: audit 2023-12-16T07:43:45.835573+0000 mon.c (mon.1) 43 : audit [INF] from='client.? 172.21.15.7:0/4171294058' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3262094376"}]: dispatch 2023-12-16T07:43:46.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:46 smithi007 bash[30436]: audit 2023-12-16T07:43:45.836757+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3262094376"}]: dispatch 2023-12-16T07:43:46.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:46 smithi007 bash[25009]: audit 2023-12-16T07:43:45.442932+0000 mon.a (mon.0) 977 : audit [INF] from='client.? 172.21.15.7:0/2161077206' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1908420919"}]': finished 2023-12-16T07:43:46.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:46 smithi007 bash[25009]: cluster 2023-12-16T07:43:45.443032+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2023-12-16T07:43:46.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:46 smithi007 bash[25009]: audit 2023-12-16T07:43:45.835573+0000 mon.c (mon.1) 43 : audit [INF] from='client.? 172.21.15.7:0/4171294058' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3262094376"}]: dispatch 2023-12-16T07:43:46.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:46 smithi007 bash[25009]: audit 2023-12-16T07:43:45.836757+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3262094376"}]: dispatch 2023-12-16T07:43:46.898 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:46 smithi169 bash[25083]: audit 2023-12-16T07:43:45.442932+0000 mon.a (mon.0) 977 : audit [INF] from='client.? 172.21.15.7:0/2161077206' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1908420919"}]': finished 2023-12-16T07:43:46.898 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:46 smithi169 bash[25083]: cluster 2023-12-16T07:43:45.443032+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2023-12-16T07:43:46.898 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:46 smithi169 bash[25083]: audit 2023-12-16T07:43:45.835573+0000 mon.c (mon.1) 43 : audit [INF] from='client.? 172.21.15.7:0/4171294058' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3262094376"}]: dispatch 2023-12-16T07:43:46.898 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:46 smithi169 bash[25083]: audit 2023-12-16T07:43:45.836757+0000 mon.a (mon.0) 979 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3262094376"}]: dispatch 2023-12-16T07:43:47.714 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:47 smithi169 bash[25083]: cluster 2023-12-16T07:43:46.003434+0000 mgr.x (mgr.24968) 63 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:47.714 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:47 smithi169 bash[25083]: audit 2023-12-16T07:43:46.463523+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3262094376"}]': finished 2023-12-16T07:43:47.714 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:47 smithi169 bash[25083]: cluster 2023-12-16T07:43:46.463624+0000 mon.a (mon.0) 981 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2023-12-16T07:43:47.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:47 smithi007 bash[30436]: cluster 2023-12-16T07:43:46.003434+0000 mgr.x (mgr.24968) 63 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:47.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:47 smithi007 bash[30436]: audit 2023-12-16T07:43:46.463523+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3262094376"}]': finished 2023-12-16T07:43:47.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:47 smithi007 bash[30436]: cluster 2023-12-16T07:43:46.463624+0000 mon.a (mon.0) 981 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2023-12-16T07:43:47.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:47 smithi007 bash[25009]: cluster 2023-12-16T07:43:46.003434+0000 mgr.x (mgr.24968) 63 : cluster [DBG] pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:43:47.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:47 smithi007 bash[25009]: audit 2023-12-16T07:43:46.463523+0000 mon.a (mon.0) 980 : audit [INF] from='client.? ' entity='client.iscsi.foo.smithi007.crrdaq' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3262094376"}]': finished 2023-12-16T07:43:47.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:47 smithi007 bash[25009]: cluster 2023-12-16T07:43:46.463624+0000 mon.a (mon.0) 981 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2023-12-16T07:43:48.121 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:43:48.885 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:43:48.885 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 starting - - - - 2023-12-16T07:43:48.885 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (7m) 40s ago 7m 47.3M - 8.3.5 dad864ee21e9 25b9cf47977e 2023-12-16T07:43:48.885 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 starting - - - - 2023-12-16T07:43:48.885 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283 running (68s) 40s ago 13m 475M - 19.0.0-191-g6d933094 a29c258b5d6f e5055e9fae4f 2023-12-16T07:43:48.885 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:9283 running (15m) 40s ago 15m 404M - 17.2.0 e1d6a67b021e 5219d4698403 2023-12-16T07:43:48.885 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (15m) 40s ago 15m 62.2M 2048M 17.2.0 e1d6a67b021e c0468ecbd44e 2023-12-16T07:43:48.885 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (13m) 40s ago 13m 49.3M 2048M 17.2.0 e1d6a67b021e b47a960c6102 2023-12-16T07:43:48.885 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (13m) 40s ago 13m 45.6M 2048M 17.2.0 e1d6a67b021e c659b986f201 2023-12-16T07:43:48.885 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 starting - - - - 2023-12-16T07:43:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (7m) 40s ago 7m 11.9M - 1.3.1 1dbe0e931976 b050faea525d 2023-12-16T07:43:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (12m) 40s ago 12m 56.6M 2782M 17.2.0 e1d6a67b021e dbc4d9538238 2023-12-16T07:43:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (12m) 40s ago 12m 58.0M 2782M 17.2.0 e1d6a67b021e 14637e0f389c 2023-12-16T07:43:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (11m) 40s ago 11m 55.6M 2782M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:43:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (10m) 40s ago 11m 54.8M 2782M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:43:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (10m) 40s ago 10m 56.1M 3037M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:43:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (9m) 40s ago 9m 54.1M 3037M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:43:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (9m) 40s ago 9m 53.8M 3037M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:43:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (8m) 40s ago 8m 53.9M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:43:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (6m) 40s ago 7m 60.2M - 2.33.4 514e6a882f6e 9351c4c7d0ba 2023-12-16T07:43:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (5m) 40s ago 5m 85.4M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:43:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (5m) 40s ago 5m 85.5M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:43:48.886 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (5m) 40s ago 5m 85.2M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:43:48.887 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (5m) 40s ago 5m 85.3M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:43:49.333 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:43:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:43:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2023-12-16T07:43:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:43:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:43:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2023-12-16T07:43:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 1 2023-12-16T07:43:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:43:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:43:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2023-12-16T07:43:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:43:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2023-12-16T07:43:49.334 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:43:49.335 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:43:49.335 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:43:49.335 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:43:49.335 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2023-12-16T07:43:49.335 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 1 2023-12-16T07:43:49.335 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:43:49.335 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:43:49.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:49 smithi169 bash[25083]: cluster 2023-12-16T07:43:48.004128+0000 mgr.x (mgr.24968) 64 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:43:49.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:49 smithi169 bash[25083]: audit 2023-12-16T07:43:48.105265+0000 mgr.x (mgr.24968) 65 : audit [DBG] from='client.15357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:49.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:49 smithi169 bash[25083]: audit 2023-12-16T07:43:48.490363+0000 mgr.x (mgr.24968) 66 : audit [DBG] from='client.15363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:49.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:49 smithi169 bash[25083]: audit 2023-12-16T07:43:49.331331+0000 mon.a (mon.0) 982 : audit [DBG] from='client.? 172.21.15.7:0/50108313' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:43:49.723 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:43:49.723 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:43:49.723 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:43:49.724 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:43:49.724 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [], 2023-12-16T07:43:49.724 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "1/23 daemons upgraded", 2023-12-16T07:43:49.724 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "", 2023-12-16T07:43:49.724 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:43:49.724 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:43:49.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:49 smithi007 bash[25009]: cluster 2023-12-16T07:43:48.004128+0000 mgr.x (mgr.24968) 64 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:43:49.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:49 smithi007 bash[25009]: audit 2023-12-16T07:43:48.105265+0000 mgr.x (mgr.24968) 65 : audit [DBG] from='client.15357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:49.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:49 smithi007 bash[25009]: audit 2023-12-16T07:43:48.490363+0000 mgr.x (mgr.24968) 66 : audit [DBG] from='client.15363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:49.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:49 smithi007 bash[25009]: audit 2023-12-16T07:43:49.331331+0000 mon.a (mon.0) 982 : audit [DBG] from='client.? 172.21.15.7:0/50108313' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:43:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:49 smithi007 bash[30436]: cluster 2023-12-16T07:43:48.004128+0000 mgr.x (mgr.24968) 64 : cluster [DBG] pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:43:49.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:49 smithi007 bash[30436]: audit 2023-12-16T07:43:48.105265+0000 mgr.x (mgr.24968) 65 : audit [DBG] from='client.15357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:49.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:49 smithi007 bash[30436]: audit 2023-12-16T07:43:48.490363+0000 mgr.x (mgr.24968) 66 : audit [DBG] from='client.15363 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:49.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:49 smithi007 bash[30436]: audit 2023-12-16T07:43:49.331331+0000 mon.a (mon.0) 982 : audit [DBG] from='client.? 172.21.15.7:0/50108313' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:43:50.178 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:43:50.646 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[25083]: audit 2023-12-16T07:43:48.860753+0000 mgr.x (mgr.24968) 67 : audit [DBG] from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:50.647 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[25083]: audit 2023-12-16T07:43:49.523190+0000 mgr.x (mgr.24968) 68 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:50.647 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[25083]: audit 2023-12-16T07:43:50.176091+0000 mon.c (mon.1) 44 : audit [DBG] from='client.? 172.21.15.7:0/2128043618' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:43:50.647 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 systemd[1]: Stopping Ceph prometheus.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:43:50.647 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[47134]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-prometheus.a 2023-12-16T07:43:50.648 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[41600]: ts=2023-12-16T07:43:50.423Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2023-12-16T07:43:50.648 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[41600]: ts=2023-12-16T07:43:50.423Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2023-12-16T07:43:50.648 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[41600]: ts=2023-12-16T07:43:50.423Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2023-12-16T07:43:50.648 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[41600]: ts=2023-12-16T07:43:50.423Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2023-12-16T07:43:50.648 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[41600]: ts=2023-12-16T07:43:50.423Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2023-12-16T07:43:50.648 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[41600]: ts=2023-12-16T07:43:50.423Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2023-12-16T07:43:50.648 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[41600]: ts=2023-12-16T07:43:50.423Z caller=main.go:828 level=info msg="Scrape manager stopped" 2023-12-16T07:43:50.648 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[41600]: ts=2023-12-16T07:43:50.424Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2023-12-16T07:43:50.649 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[41600]: ts=2023-12-16T07:43:50.424Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2023-12-16T07:43:50.649 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[41600]: ts=2023-12-16T07:43:50.424Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2023-12-16T07:43:50.649 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[41600]: ts=2023-12-16T07:43:50.424Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2023-12-16T07:43:50.649 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[41600]: ts=2023-12-16T07:43:50.425Z caller=main.go:1066 level=info msg="See you next time!" 2023-12-16T07:43:50.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:50 smithi007 bash[25009]: audit 2023-12-16T07:43:48.860753+0000 mgr.x (mgr.24968) 67 : audit [DBG] from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:50.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:50 smithi007 bash[25009]: audit 2023-12-16T07:43:49.523190+0000 mgr.x (mgr.24968) 68 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:50.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:50 smithi007 bash[25009]: audit 2023-12-16T07:43:50.176091+0000 mon.c (mon.1) 44 : audit [DBG] from='client.? 172.21.15.7:0/2128043618' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:43:50.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:50 smithi007 bash[30436]: audit 2023-12-16T07:43:48.860753+0000 mgr.x (mgr.24968) 67 : audit [DBG] from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:50.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:50 smithi007 bash[30436]: audit 2023-12-16T07:43:49.523190+0000 mgr.x (mgr.24968) 68 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:43:50.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:50 smithi007 bash[30436]: audit 2023-12-16T07:43:50.176091+0000 mon.c (mon.1) 44 : audit [DBG] from='client.? 172.21.15.7:0/2128043618' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:43:51.205 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[47143]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-prometheus-a 2023-12-16T07:43:51.205 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:50 smithi169 bash[47180]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-prometheus.a 2023-12-16T07:43:51.205 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:51 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@prometheus.a.service: Succeeded. 2023-12-16T07:43:51.205 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:51 smithi169 systemd[1]: Stopped Ceph prometheus.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:43:51.205 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:51 smithi169 systemd[1]: Started Ceph prometheus.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:43:51.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:51 smithi007 bash[25009]: audit 2023-12-16T07:43:49.722272+0000 mgr.x (mgr.24968) 69 : audit [DBG] from='client.25288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:51.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:51 smithi007 bash[25009]: cluster 2023-12-16T07:43:50.005073+0000 mgr.x (mgr.24968) 70 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-12-16T07:43:51.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:51 smithi007 bash[25009]: audit 2023-12-16T07:43:51.196966+0000 mon.a (mon.0) 983 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:51.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:51 smithi007 bash[25009]: audit 2023-12-16T07:43:51.208221+0000 mon.a (mon.0) 984 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:51.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:51 smithi007 bash[30436]: audit 2023-12-16T07:43:49.722272+0000 mgr.x (mgr.24968) 69 : audit [DBG] from='client.25288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:51.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:51 smithi007 bash[30436]: cluster 2023-12-16T07:43:50.005073+0000 mgr.x (mgr.24968) 70 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-12-16T07:43:51.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:51 smithi007 bash[30436]: audit 2023-12-16T07:43:51.196966+0000 mon.a (mon.0) 983 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:51.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:51 smithi007 bash[30436]: audit 2023-12-16T07:43:51.208221+0000 mon.a (mon.0) 984 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:51.891 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:51 smithi169 bash[25083]: audit 2023-12-16T07:43:49.722272+0000 mgr.x (mgr.24968) 69 : audit [DBG] from='client.25288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:43:51.891 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:51 smithi169 bash[25083]: cluster 2023-12-16T07:43:50.005073+0000 mgr.x (mgr.24968) 70 : cluster [DBG] pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2023-12-16T07:43:51.891 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:51 smithi169 bash[25083]: audit 2023-12-16T07:43:51.196966+0000 mon.a (mon.0) 983 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:51.891 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:51 smithi169 bash[25083]: audit 2023-12-16T07:43:51.208221+0000 mon.a (mon.0) 984 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:51.892 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:51 smithi169 systemd[1]: Stopping Ceph node-exporter.b for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:43:51.893 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:51 smithi169 bash[47231]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-node-exporter.b 2023-12-16T07:43:52.798 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[25083]: cephadm 2023-12-16T07:43:51.214490+0000 mgr.x (mgr.24968) 71 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2023-12-16T07:43:52.798 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[25083]: cephadm 2023-12-16T07:43:51.215055+0000 mgr.x (mgr.24968) 72 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi169 2023-12-16T07:43:52.798 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47241]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-node-exporter-b 2023-12-16T07:43:52.798 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47318]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-node-exporter.b 2023-12-16T07:43:52.798 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47208]: ts=2023-12-16T07:43:52.653Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2023-12-16T07:43:52.799 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47208]: ts=2023-12-16T07:43:52.653Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2023-12-16T07:43:52.799 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47208]: ts=2023-12-16T07:43:52.653Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2023-12-16T07:43:52.799 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47208]: ts=2023-12-16T07:43:52.654Z caller=main.go:518 level=info host_details="(Linux 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 smithi169 (none))" 2023-12-16T07:43:52.799 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47208]: ts=2023-12-16T07:43:52.654Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2023-12-16T07:43:52.799 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47208]: ts=2023-12-16T07:43:52.654Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2023-12-16T07:43:52.799 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47208]: ts=2023-12-16T07:43:52.656Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2023-12-16T07:43:52.799 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47208]: ts=2023-12-16T07:43:52.657Z caller=main.go:923 level=info msg="Starting TSDB ..." 2023-12-16T07:43:52.799 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47208]: ts=2023-12-16T07:43:52.661Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2023-12-16T07:43:52.799 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47208]: ts=2023-12-16T07:43:52.664Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2023-12-16T07:43:52.799 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47208]: ts=2023-12-16T07:43:52.664Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.544µs 2023-12-16T07:43:52.799 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47208]: ts=2023-12-16T07:43:52.664Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2023-12-16T07:43:52.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:52 smithi007 bash[25009]: cephadm 2023-12-16T07:43:51.214490+0000 mgr.x (mgr.24968) 71 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2023-12-16T07:43:52.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:52 smithi007 bash[25009]: cephadm 2023-12-16T07:43:51.215055+0000 mgr.x (mgr.24968) 72 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi169 2023-12-16T07:43:52.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:52 smithi007 bash[30436]: cephadm 2023-12-16T07:43:51.214490+0000 mgr.x (mgr.24968) 71 : cephadm [INF] Reconfiguring node-exporter.b (dependencies changed)... 2023-12-16T07:43:52.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:52 smithi007 bash[30436]: cephadm 2023-12-16T07:43:51.215055+0000 mgr.x (mgr.24968) 72 : cephadm [INF] Reconfiguring daemon node-exporter.b on smithi169 2023-12-16T07:43:52.833 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:52 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2023-12-16T07:43:52.833 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:52 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2023-12-16T07:43:52.833 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:52 smithi169 systemd[1]: Stopped Ceph node-exporter.b for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:43:52.833 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:52 smithi169 systemd[1]: Started Ceph node-exporter.b for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:43:53.108 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:52 smithi169 bash[47208]: ts=2023-12-16T07:43:52.832Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2023-12-16T07:43:53.145 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:53 smithi169 bash[47208]: ts=2023-12-16T07:43:53.108Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2023-12-16T07:43:53.145 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:53 smithi169 bash[47208]: ts=2023-12-16T07:43:53.108Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2023-12-16T07:43:53.145 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:53 smithi169 bash[47208]: ts=2023-12-16T07:43:53.108Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=82.568µs wal_replay_duration=444.129895ms total_replay_duration=444.300939ms 2023-12-16T07:43:53.146 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:53 smithi169 bash[47208]: ts=2023-12-16T07:43:53.115Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2023-12-16T07:43:53.146 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:53 smithi169 bash[47208]: ts=2023-12-16T07:43:53.115Z caller=main.go:947 level=info msg="TSDB started" 2023-12-16T07:43:53.146 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:53 smithi169 bash[47208]: ts=2023-12-16T07:43:53.115Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-12-16T07:43:53.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:53 smithi169 bash[47208]: ts=2023-12-16T07:43:53.145Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=30.15443ms db_storage=1.22µs remote_storage=1.824µs web_handler=626ns query_engine=1.026µs scrape=472.464µs scrape_sd=167.212µs notify=20.667µs notify_sd=48.409µs rules=28.866778ms 2023-12-16T07:43:53.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:43:53 smithi169 bash[47208]: ts=2023-12-16T07:43:53.145Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2023-12-16T07:43:53.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:53 smithi007 bash[25009]: cluster 2023-12-16T07:43:52.006817+0000 mgr.x (mgr.24968) 73 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:43:53.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:53 smithi007 bash[25009]: audit 2023-12-16T07:43:52.850342+0000 mon.a (mon.0) 985 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:53.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:53 smithi007 bash[25009]: audit 2023-12-16T07:43:52.862124+0000 mon.a (mon.0) 986 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:53.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:53 smithi007 bash[25009]: audit 2023-12-16T07:43:52.978137+0000 mon.a (mon.0) 987 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:53.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:53 smithi007 bash[25009]: audit 2023-12-16T07:43:52.986572+0000 mon.a (mon.0) 988 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:53.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:53 smithi007 bash[25009]: audit 2023-12-16T07:43:52.993132+0000 mon.b (mon.2) 64 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T07:43:53.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:53 smithi007 bash[30436]: cluster 2023-12-16T07:43:52.006817+0000 mgr.x (mgr.24968) 73 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:43:53.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:53 smithi007 bash[30436]: audit 2023-12-16T07:43:52.850342+0000 mon.a (mon.0) 985 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:53.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:53 smithi007 bash[30436]: audit 2023-12-16T07:43:52.862124+0000 mon.a (mon.0) 986 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:53.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:53 smithi007 bash[30436]: audit 2023-12-16T07:43:52.978137+0000 mon.a (mon.0) 987 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:53.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:53 smithi007 bash[30436]: audit 2023-12-16T07:43:52.986572+0000 mon.a (mon.0) 988 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:53.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:53 smithi007 bash[30436]: audit 2023-12-16T07:43:52.993132+0000 mon.b (mon.2) 64 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T07:43:53.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:53 smithi169 bash[25083]: cluster 2023-12-16T07:43:52.006817+0000 mgr.x (mgr.24968) 73 : cluster [DBG] pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:43:53.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:53 smithi169 bash[25083]: audit 2023-12-16T07:43:52.850342+0000 mon.a (mon.0) 985 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:53.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:53 smithi169 bash[25083]: audit 2023-12-16T07:43:52.862124+0000 mon.a (mon.0) 986 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:53.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:53 smithi169 bash[25083]: audit 2023-12-16T07:43:52.978137+0000 mon.a (mon.0) 987 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:53.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:53 smithi169 bash[25083]: audit 2023-12-16T07:43:52.986572+0000 mon.a (mon.0) 988 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:43:53.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:53 smithi169 bash[25083]: audit 2023-12-16T07:43:52.993132+0000 mon.b (mon.2) 64 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T07:43:54.434 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.152Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2023-12-16T07:43:54.434 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.152Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2023-12-16T07:43:54.434 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.153Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2023-12-16T07:43:54.434 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.153Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2023-12-16T07:43:54.434 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2023-12-16T07:43:54.434 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=arp 2023-12-16T07:43:54.434 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=bcache 2023-12-16T07:43:54.434 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=bonding 2023-12-16T07:43:54.434 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=btrfs 2023-12-16T07:43:54.434 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=conntrack 2023-12-16T07:43:54.434 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=cpu 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=cpufreq 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=diskstats 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=dmi 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=edac 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=entropy 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=fibrechannel 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=filefd 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=filesystem 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=hwmon 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=infiniband 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=ipvs 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=loadavg 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=mdadm 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=meminfo 2023-12-16T07:43:54.435 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=netclass 2023-12-16T07:43:54.436 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=netdev 2023-12-16T07:43:54.436 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=netstat 2023-12-16T07:43:54.436 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=nfs 2023-12-16T07:43:54.436 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=nfsd 2023-12-16T07:43:54.436 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=nvme 2023-12-16T07:43:54.436 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=os 2023-12-16T07:43:54.436 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2023-12-16T07:43:54.436 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=pressure 2023-12-16T07:43:54.436 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=rapl 2023-12-16T07:43:54.437 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=schedstat 2023-12-16T07:43:54.437 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=sockstat 2023-12-16T07:43:54.437 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=softnet 2023-12-16T07:43:54.437 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=stat 2023-12-16T07:43:54.437 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=tapestats 2023-12-16T07:43:54.437 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=textfile 2023-12-16T07:43:54.437 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=thermal_zone 2023-12-16T07:43:54.437 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=time 2023-12-16T07:43:54.437 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=udp_queues 2023-12-16T07:43:54.437 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=uname 2023-12-16T07:43:54.437 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=vmstat 2023-12-16T07:43:54.437 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=xfs 2023-12-16T07:43:54.437 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:115 level=info collector=zfs 2023-12-16T07:43:54.437 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.155Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2023-12-16T07:43:54.438 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[47366]: ts=2023-12-16T07:43:54.156Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2023-12-16T07:43:54.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:54 smithi007 bash[25009]: cephadm 2023-12-16T07:43:52.883464+0000 mgr.x (mgr.24968) 74 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2023-12-16T07:43:54.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:54 smithi007 bash[25009]: cephadm 2023-12-16T07:43:52.900846+0000 mgr.x (mgr.24968) 75 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2023-12-16T07:43:54.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:54 smithi007 bash[25009]: audit 2023-12-16T07:43:52.993784+0000 mgr.x (mgr.24968) 76 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T07:43:54.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:54 smithi007 bash[25009]: cephadm 2023-12-16T07:43:52.997049+0000 mgr.x (mgr.24968) 77 : cephadm [INF] Reconfiguring daemon grafana.a on smithi169 2023-12-16T07:43:54.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:54 smithi007 bash[25009]: audit 2023-12-16T07:43:54.169235+0000 mon.b (mon.2) 65 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:43:54.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:54 smithi007 bash[30436]: cephadm 2023-12-16T07:43:52.883464+0000 mgr.x (mgr.24968) 74 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2023-12-16T07:43:54.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:54 smithi007 bash[30436]: cephadm 2023-12-16T07:43:52.900846+0000 mgr.x (mgr.24968) 75 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2023-12-16T07:43:54.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:54 smithi007 bash[30436]: audit 2023-12-16T07:43:52.993784+0000 mgr.x (mgr.24968) 76 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T07:43:54.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:54 smithi007 bash[30436]: cephadm 2023-12-16T07:43:52.997049+0000 mgr.x (mgr.24968) 77 : cephadm [INF] Reconfiguring daemon grafana.a on smithi169 2023-12-16T07:43:54.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:54 smithi007 bash[30436]: audit 2023-12-16T07:43:54.169235+0000 mon.b (mon.2) 65 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:43:54.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[25083]: cephadm 2023-12-16T07:43:52.883464+0000 mgr.x (mgr.24968) 74 : cephadm [INF] Reconfiguring grafana.a (dependencies changed)... 2023-12-16T07:43:54.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[25083]: cephadm 2023-12-16T07:43:52.900846+0000 mgr.x (mgr.24968) 75 : cephadm [INF] Regenerating cephadm self-signed grafana TLS certificates 2023-12-16T07:43:54.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[25083]: audit 2023-12-16T07:43:52.993784+0000 mgr.x (mgr.24968) 76 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-16T07:43:54.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[25083]: cephadm 2023-12-16T07:43:52.997049+0000 mgr.x (mgr.24968) 77 : cephadm [INF] Reconfiguring daemon grafana.a on smithi169 2023-12-16T07:43:54.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:54 smithi169 bash[25083]: audit 2023-12-16T07:43:54.169235+0000 mon.b (mon.2) 65 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:43:55.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:55 smithi007 bash[25009]: cluster 2023-12-16T07:43:54.007631+0000 mgr.x (mgr.24968) 78 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2023-12-16T07:43:55.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:55 smithi007 bash[30436]: cluster 2023-12-16T07:43:54.007631+0000 mgr.x (mgr.24968) 78 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2023-12-16T07:43:55.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:55 smithi169 bash[25083]: cluster 2023-12-16T07:43:54.007631+0000 mgr.x (mgr.24968) 78 : cluster [DBG] pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2023-12-16T07:43:57.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:57 smithi007 bash[25009]: cluster 2023-12-16T07:43:56.008830+0000 mgr.x (mgr.24968) 79 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:43:57.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:57 smithi007 bash[30436]: cluster 2023-12-16T07:43:56.008830+0000 mgr.x (mgr.24968) 79 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:43:57.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:57 smithi169 bash[25083]: cluster 2023-12-16T07:43:56.008830+0000 mgr.x (mgr.24968) 79 : cluster [DBG] pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:43:59.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:43:58 smithi169 bash[45188]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:43:58] "GET /metrics HTTP/1.1" 200 38129 "" "Prometheus/2.33.4" 2023-12-16T07:43:59.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:43:59 smithi007 bash[30436]: cluster 2023-12-16T07:43:58.009476+0000 mgr.x (mgr.24968) 80 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2023-12-16T07:43:59.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:43:59 smithi007 bash[25009]: cluster 2023-12-16T07:43:58.009476+0000 mgr.x (mgr.24968) 80 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2023-12-16T07:43:59.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:43:59 smithi169 bash[25083]: cluster 2023-12-16T07:43:58.009476+0000 mgr.x (mgr.24968) 80 : cluster [DBG] pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2023-12-16T07:44:00.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:00 smithi007 bash[25009]: audit 2023-12-16T07:43:59.532850+0000 mgr.x (mgr.24968) 81 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:00.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:00 smithi007 bash[30436]: audit 2023-12-16T07:43:59.532850+0000 mgr.x (mgr.24968) 81 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:00.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:00 smithi169 bash[25083]: audit 2023-12-16T07:43:59.532850+0000 mgr.x (mgr.24968) 81 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:01.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:01 smithi007 bash[25009]: cluster 2023-12-16T07:44:00.010110+0000 mgr.x (mgr.24968) 82 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:01.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:01 smithi007 bash[30436]: cluster 2023-12-16T07:44:00.010110+0000 mgr.x (mgr.24968) 82 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:01.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:01 smithi169 bash[25083]: cluster 2023-12-16T07:44:00.010110+0000 mgr.x (mgr.24968) 82 : cluster [DBG] pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:03.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:03 smithi007 bash[25009]: cluster 2023-12-16T07:44:02.011355+0000 mgr.x (mgr.24968) 83 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:03.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:03 smithi007 bash[30436]: cluster 2023-12-16T07:44:02.011355+0000 mgr.x (mgr.24968) 83 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:03.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:03 smithi169 bash[25083]: cluster 2023-12-16T07:44:02.011355+0000 mgr.x (mgr.24968) 83 : cluster [DBG] pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:05.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:05 smithi007 bash[25009]: cluster 2023-12-16T07:44:04.011975+0000 mgr.x (mgr.24968) 84 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:05.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:05 smithi007 bash[30436]: cluster 2023-12-16T07:44:04.011975+0000 mgr.x (mgr.24968) 84 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:05.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:05 smithi169 bash[25083]: cluster 2023-12-16T07:44:04.011975+0000 mgr.x (mgr.24968) 84 : cluster [DBG] pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:07.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:07 smithi007 bash[25009]: cluster 2023-12-16T07:44:06.013661+0000 mgr.x (mgr.24968) 85 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:07.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:07 smithi007 bash[30436]: cluster 2023-12-16T07:44:06.013661+0000 mgr.x (mgr.24968) 85 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:07.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:07 smithi169 bash[25083]: cluster 2023-12-16T07:44:06.013661+0000 mgr.x (mgr.24968) 85 : cluster [DBG] pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:08.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:08 smithi169 bash[45188]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:44:08] "GET /metrics HTTP/1.1" 200 38129 "" "Prometheus/2.33.4" 2023-12-16T07:44:08.934 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:44:08 smithi169 bash[47208]: ts=2023-12-16T07:44:08.662Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:44:09.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:09 smithi007 bash[25009]: cluster 2023-12-16T07:44:08.014540+0000 mgr.x (mgr.24968) 86 : cluster [DBG] pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:09.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:09 smithi007 bash[25009]: audit 2023-12-16T07:44:09.169997+0000 mon.b (mon.2) 66 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:09.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:09 smithi007 bash[30436]: cluster 2023-12-16T07:44:08.014540+0000 mgr.x (mgr.24968) 86 : cluster [DBG] pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:09.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:09 smithi007 bash[30436]: audit 2023-12-16T07:44:09.169997+0000 mon.b (mon.2) 66 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:09.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:09 smithi169 bash[25083]: cluster 2023-12-16T07:44:08.014540+0000 mgr.x (mgr.24968) 86 : cluster [DBG] pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:09.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:09 smithi169 bash[25083]: audit 2023-12-16T07:44:09.169997+0000 mon.b (mon.2) 66 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:10.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:10 smithi007 bash[25009]: audit 2023-12-16T07:44:09.541006+0000 mgr.x (mgr.24968) 87 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:10.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:10 smithi007 bash[30436]: audit 2023-12-16T07:44:09.541006+0000 mgr.x (mgr.24968) 87 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:10.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:10 smithi169 bash[25083]: audit 2023-12-16T07:44:09.541006+0000 mgr.x (mgr.24968) 87 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:11.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:11 smithi007 bash[30436]: cluster 2023-12-16T07:44:10.015317+0000 mgr.x (mgr.24968) 88 : cluster [DBG] pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:11.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:11 smithi007 bash[25009]: cluster 2023-12-16T07:44:10.015317+0000 mgr.x (mgr.24968) 88 : cluster [DBG] pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:11 smithi169 bash[25083]: cluster 2023-12-16T07:44:10.015317+0000 mgr.x (mgr.24968) 88 : cluster [DBG] pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:13.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:13 smithi007 bash[30436]: cluster 2023-12-16T07:44:12.016916+0000 mgr.x (mgr.24968) 89 : cluster [DBG] pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:13.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:13 smithi007 bash[25009]: cluster 2023-12-16T07:44:12.016916+0000 mgr.x (mgr.24968) 89 : cluster [DBG] pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:13.824 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:13 smithi169 bash[25083]: cluster 2023-12-16T07:44:12.016916+0000 mgr.x (mgr.24968) 89 : cluster [DBG] pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:15 smithi169 bash[25083]: cluster 2023-12-16T07:44:14.017711+0000 mgr.x (mgr.24968) 90 : cluster [DBG] pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:15.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:15 smithi007 bash[30436]: cluster 2023-12-16T07:44:14.017711+0000 mgr.x (mgr.24968) 90 : cluster [DBG] pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:15.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:15 smithi007 bash[25009]: cluster 2023-12-16T07:44:14.017711+0000 mgr.x (mgr.24968) 90 : cluster [DBG] pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:16.601 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:16 smithi169 systemd[1]: Stopping Ceph grafana.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:44:16.601 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:16 smithi169 bash[47613]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-grafana.a 2023-12-16T07:44:16.601 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:16 smithi169 bash[40964]: t=2023-12-16T07:44:16+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2023-12-16T07:44:16.601 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:16 smithi169 bash[40964]: t=2023-12-16T07:44:16+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 2023-12-16T07:44:17.146 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:16 smithi169 bash[47622]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-grafana-a 2023-12-16T07:44:17.146 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:16 smithi169 bash[47661]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-grafana.a 2023-12-16T07:44:17.146 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:17 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@grafana.a.service: Succeeded. 2023-12-16T07:44:17.146 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:17 smithi169 systemd[1]: Stopped Ceph grafana.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:44:17.146 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:17 smithi169 systemd[1]: Started Ceph grafana.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:44:17.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Bus STOPPING 2023-12-16T07:44:17.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2023-12-16T07:44:17.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Bus STOPPED 2023-12-16T07:44:17.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Bus STARTING 2023-12-16T07:44:17.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Serving on http://:::9283 2023-12-16T07:44:17.434 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Bus STARTED 2023-12-16T07:44:17.434 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Bus STOPPING 2023-12-16T07:44:17.434 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2023-12-16T07:44:17.434 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Bus STOPPED 2023-12-16T07:44:17.434 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Bus STARTING 2023-12-16T07:44:17.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: cluster 2023-12-16T07:44:16.019410+0000 mgr.x (mgr.24968) 91 : cluster [DBG] pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:17.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.130597+0000 mon.a (mon.0) 989 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.141821+0000 mon.a (mon.0) 990 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.183419+0000 mon.b (mon.2) 67 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T07:44:17.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.186504+0000 mon.b (mon.2) 68 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2023-12-16T07:44:17.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.196321+0000 mon.a (mon.0) 991 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.212368+0000 mon.b (mon.2) 69 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:44:17.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.236057+0000 mon.a (mon.0) 992 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.241613+0000 mon.b (mon.2) 70 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:44:17.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.245806+0000 mon.b (mon.2) 71 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:44:17.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.252379+0000 mon.a (mon.0) 993 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.261493+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:44:17.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.264277+0000 mon.b (mon.2) 73 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi169.front.sepia.ceph.com:9095"}]: dispatch 2023-12-16T07:44:17.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.271926+0000 mon.a (mon.0) 994 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.290421+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T07:44:17.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.297793+0000 mon.b (mon.2) 75 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi169.front.sepia.ceph.com:3000"}]: dispatch 2023-12-16T07:44:17.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.307018+0000 mon.a (mon.0) 995 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[25009]: audit 2023-12-16T07:44:17.336488+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:44:17.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: cluster 2023-12-16T07:44:16.019410+0000 mgr.x (mgr.24968) 91 : cluster [DBG] pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:17.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.130597+0000 mon.a (mon.0) 989 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.141821+0000 mon.a (mon.0) 990 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.183419+0000 mon.b (mon.2) 67 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T07:44:17.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.186504+0000 mon.b (mon.2) 68 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2023-12-16T07:44:17.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.196321+0000 mon.a (mon.0) 991 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.212368+0000 mon.b (mon.2) 69 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:44:17.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.236057+0000 mon.a (mon.0) 992 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.241613+0000 mon.b (mon.2) 70 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:44:17.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.245806+0000 mon.b (mon.2) 71 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:44:17.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.252379+0000 mon.a (mon.0) 993 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.261493+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:44:17.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.264277+0000 mon.b (mon.2) 73 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi169.front.sepia.ceph.com:9095"}]: dispatch 2023-12-16T07:44:17.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.271926+0000 mon.a (mon.0) 994 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.290421+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T07:44:17.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.297793+0000 mon.b (mon.2) 75 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi169.front.sepia.ceph.com:3000"}]: dispatch 2023-12-16T07:44:17.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.307018+0000 mon.a (mon.0) 995 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:17 smithi007 bash[30436]: audit 2023-12-16T07:44:17.336488+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:44:17.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Serving on http://:::9283 2023-12-16T07:44:17.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Bus STARTED 2023-12-16T07:44:17.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Bus STOPPING 2023-12-16T07:44:17.934 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2023-12-16T07:44:17.934 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Bus STOPPED 2023-12-16T07:44:17.934 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Bus STARTING 2023-12-16T07:44:17.934 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Serving on http://:::9283 2023-12-16T07:44:17.934 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[45188]: [16/Dec/2023:07:44:17] ENGINE Bus STARTED 2023-12-16T07:44:17.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: cluster 2023-12-16T07:44:16.019410+0000 mgr.x (mgr.24968) 91 : cluster [DBG] pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:17.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.130597+0000 mon.a (mon.0) 989 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.141821+0000 mon.a (mon.0) 990 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.183419+0000 mon.b (mon.2) 67 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T07:44:17.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.186504+0000 mon.b (mon.2) 68 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2023-12-16T07:44:17.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.196321+0000 mon.a (mon.0) 991 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.212368+0000 mon.b (mon.2) 69 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:44:17.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.236057+0000 mon.a (mon.0) 992 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.241613+0000 mon.b (mon.2) 70 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:44:17.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.245806+0000 mon.b (mon.2) 71 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:44:17.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.252379+0000 mon.a (mon.0) 993 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.261493+0000 mon.b (mon.2) 72 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:44:17.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.264277+0000 mon.b (mon.2) 73 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi169.front.sepia.ceph.com:9095"}]: dispatch 2023-12-16T07:44:17.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.271926+0000 mon.a (mon.0) 994 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.290421+0000 mon.b (mon.2) 74 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T07:44:17.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.297793+0000 mon.b (mon.2) 75 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi169.front.sepia.ceph.com:3000"}]: dispatch 2023-12-16T07:44:17.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.307018+0000 mon.a (mon.0) 995 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:17.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:17 smithi169 bash[25083]: audit 2023-12-16T07:44:17.336488+0000 mon.b (mon.2) 76 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:44:18.271 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2023-12-16T07:44:18.271 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2023-12-16T07:44:18.271 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2023-12-16T07:44:18.271 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2023-12-16T07:44:18.271 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2023-12-16T07:44:18.271 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2023-12-16T07:44:18.271 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2023-12-16T07:44:18.271 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2023-12-16T07:44:18.271 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2023-12-16T07:44:18.271 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2023-12-16T07:44:18.272 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2023-12-16T07:44:18.272 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2023-12-16T07:44:18.272 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2023-12-16T07:44:18.272 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="App mode production" logger=settings 2023-12-16T07:44:18.272 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2023-12-16T07:44:18.272 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2023-12-16T07:44:18.272 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Starting DB migrations" logger=migrator 2023-12-16T07:44:18.272 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=616.917µs 2023-12-16T07:44:18.272 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Created default organization" logger=sqlstore 2023-12-16T07:44:18.272 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2023-12-16T07:44:18.272 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2023-12-16T07:44:18.272 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2023-12-16T07:44:18.272 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2023-12-16T07:44:18.530 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2023-12-16T07:44:18.530 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2023-12-16T07:44:18.531 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2023-12-16T07:44:18.531 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2023-12-16T07:44:18.531 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="warming cache for startup" logger=ngalert 2023-12-16T07:44:18.807 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47208]: ts=2023-12-16T07:44:18.662Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:44:18.807 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2023-12-16T07:44:18.807 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[47688]: t=2023-12-16T07:44:18+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2023-12-16T07:44:18.807 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[25083]: audit 2023-12-16T07:44:17.184348+0000 mgr.x (mgr.24968) 92 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T07:44:18.807 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[25083]: audit 2023-12-16T07:44:17.187271+0000 mgr.x (mgr.24968) 93 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2023-12-16T07:44:18.808 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[25083]: audit 2023-12-16T07:44:17.213587+0000 mgr.x (mgr.24968) 94 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:44:18.808 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[25083]: cephadm 2023-12-16T07:44:17.241023+0000 mgr.x (mgr.24968) 95 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2023-12-16T07:44:18.808 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[25083]: audit 2023-12-16T07:44:17.242461+0000 mgr.x (mgr.24968) 96 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:44:18.808 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[25083]: audit 2023-12-16T07:44:17.246681+0000 mgr.x (mgr.24968) 97 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:44:18.808 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[25083]: audit 2023-12-16T07:44:17.262371+0000 mgr.x (mgr.24968) 98 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:44:18.808 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[25083]: audit 2023-12-16T07:44:17.265144+0000 mgr.x (mgr.24968) 99 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi169.front.sepia.ceph.com:9095"}]: dispatch 2023-12-16T07:44:18.808 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[25083]: audit 2023-12-16T07:44:17.291159+0000 mgr.x (mgr.24968) 100 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T07:44:18.808 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[25083]: audit 2023-12-16T07:44:17.298613+0000 mgr.x (mgr.24968) 101 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi169.front.sepia.ceph.com:3000"}]: dispatch 2023-12-16T07:44:18.808 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[25083]: cephadm 2023-12-16T07:44:17.338884+0000 mgr.x (mgr.24968) 102 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-12-16T07:44:18.808 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[25083]: cephadm 2023-12-16T07:44:17.339709+0000 mgr.x (mgr.24968) 103 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-12-16T07:44:18.846 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[30436]: audit 2023-12-16T07:44:17.184348+0000 mgr.x (mgr.24968) 92 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T07:44:18.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[30436]: audit 2023-12-16T07:44:17.187271+0000 mgr.x (mgr.24968) 93 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2023-12-16T07:44:18.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[30436]: audit 2023-12-16T07:44:17.213587+0000 mgr.x (mgr.24968) 94 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:44:18.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[30436]: cephadm 2023-12-16T07:44:17.241023+0000 mgr.x (mgr.24968) 95 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2023-12-16T07:44:18.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[30436]: audit 2023-12-16T07:44:17.242461+0000 mgr.x (mgr.24968) 96 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:44:18.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[30436]: audit 2023-12-16T07:44:17.246681+0000 mgr.x (mgr.24968) 97 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:44:18.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[30436]: audit 2023-12-16T07:44:17.262371+0000 mgr.x (mgr.24968) 98 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:44:18.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[30436]: audit 2023-12-16T07:44:17.265144+0000 mgr.x (mgr.24968) 99 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi169.front.sepia.ceph.com:9095"}]: dispatch 2023-12-16T07:44:18.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[30436]: audit 2023-12-16T07:44:17.291159+0000 mgr.x (mgr.24968) 100 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T07:44:18.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[30436]: audit 2023-12-16T07:44:17.298613+0000 mgr.x (mgr.24968) 101 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi169.front.sepia.ceph.com:3000"}]: dispatch 2023-12-16T07:44:18.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[30436]: cephadm 2023-12-16T07:44:17.338884+0000 mgr.x (mgr.24968) 102 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-12-16T07:44:18.847 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[30436]: cephadm 2023-12-16T07:44:17.339709+0000 mgr.x (mgr.24968) 103 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-12-16T07:44:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[25009]: audit 2023-12-16T07:44:17.184348+0000 mgr.x (mgr.24968) 92 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2023-12-16T07:44:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[25009]: audit 2023-12-16T07:44:17.187271+0000 mgr.x (mgr.24968) 93 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2023-12-16T07:44:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[25009]: audit 2023-12-16T07:44:17.213587+0000 mgr.x (mgr.24968) 94 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:44:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[25009]: cephadm 2023-12-16T07:44:17.241023+0000 mgr.x (mgr.24968) 95 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2023-12-16T07:44:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[25009]: audit 2023-12-16T07:44:17.242461+0000 mgr.x (mgr.24968) 96 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:44:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[25009]: audit 2023-12-16T07:44:17.246681+0000 mgr.x (mgr.24968) 97 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:44:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[25009]: audit 2023-12-16T07:44:17.262371+0000 mgr.x (mgr.24968) 98 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:44:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[25009]: audit 2023-12-16T07:44:17.265144+0000 mgr.x (mgr.24968) 99 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi169.front.sepia.ceph.com:9095"}]: dispatch 2023-12-16T07:44:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[25009]: audit 2023-12-16T07:44:17.291159+0000 mgr.x (mgr.24968) 100 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2023-12-16T07:44:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[25009]: audit 2023-12-16T07:44:17.298613+0000 mgr.x (mgr.24968) 101 : audit [DBG] from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi169.front.sepia.ceph.com:3000"}]: dispatch 2023-12-16T07:44:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[25009]: cephadm 2023-12-16T07:44:17.338884+0000 mgr.x (mgr.24968) 102 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-12-16T07:44:18.848 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:18 smithi007 bash[25009]: cephadm 2023-12-16T07:44:17.339709+0000 mgr.x (mgr.24968) 103 : cephadm [INF] Upgrade: Need to upgrade myself (mgr.x) 2023-12-16T07:44:19.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:18 smithi169 bash[45188]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:44:18] "GET /metrics HTTP/1.1" 200 38128 "" "Prometheus/2.33.4" 2023-12-16T07:44:19.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:19 smithi007 bash[30436]: cluster 2023-12-16T07:44:18.020041+0000 mgr.x (mgr.24968) 104 : cluster [DBG] pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:19.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:19 smithi007 bash[25009]: cluster 2023-12-16T07:44:18.020041+0000 mgr.x (mgr.24968) 104 : cluster [DBG] pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:19.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:19 smithi169 bash[25083]: cluster 2023-12-16T07:44:18.020041+0000 mgr.x (mgr.24968) 104 : cluster [DBG] pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:20.584 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:44:20.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:20 smithi169 bash[25083]: audit 2023-12-16T07:44:19.549078+0000 mgr.x (mgr.24968) 105 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:20 smithi169 bash[25083]: audit 2023-12-16T07:44:19.768809+0000 mon.a (mon.0) 996 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:20 smithi169 bash[25083]: audit 2023-12-16T07:44:19.772089+0000 mon.a (mon.0) 997 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:44:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:20 smithi169 bash[25083]: audit 2023-12-16T07:44:19.773147+0000 mon.b (mon.2) 77 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:44:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:20 smithi169 bash[25083]: audit 2023-12-16T07:44:19.776304+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:44:20.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:20 smithi169 bash[25083]: audit 2023-12-16T07:44:19.778379+0000 mon.b (mon.2) 79 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:44:21.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:20 smithi007 bash[30436]: audit 2023-12-16T07:44:19.549078+0000 mgr.x (mgr.24968) 105 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:21.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:20 smithi007 bash[30436]: audit 2023-12-16T07:44:19.768809+0000 mon.a (mon.0) 996 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:21.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:20 smithi007 bash[30436]: audit 2023-12-16T07:44:19.772089+0000 mon.a (mon.0) 997 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:44:21.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:20 smithi007 bash[30436]: audit 2023-12-16T07:44:19.773147+0000 mon.b (mon.2) 77 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:44:21.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:20 smithi007 bash[30436]: audit 2023-12-16T07:44:19.776304+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:44:21.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:20 smithi007 bash[30436]: audit 2023-12-16T07:44:19.778379+0000 mon.b (mon.2) 79 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:44:21.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:20 smithi007 bash[25009]: audit 2023-12-16T07:44:19.549078+0000 mgr.x (mgr.24968) 105 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:21.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:20 smithi007 bash[25009]: audit 2023-12-16T07:44:19.768809+0000 mon.a (mon.0) 996 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:21.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:20 smithi007 bash[25009]: audit 2023-12-16T07:44:19.772089+0000 mon.a (mon.0) 997 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:44:21.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:20 smithi007 bash[25009]: audit 2023-12-16T07:44:19.773147+0000 mon.b (mon.2) 77 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:44:21.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:20 smithi007 bash[25009]: audit 2023-12-16T07:44:19.776304+0000 mon.b (mon.2) 78 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:44:21.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:20 smithi007 bash[25009]: audit 2023-12-16T07:44:19.778379+0000 mon.b (mon.2) 79 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:44:21.357 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:44:21.357 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 starting - - - - 2023-12-16T07:44:21.357 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 starting - - - - 2023-12-16T07:44:21.357 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 starting - - - - 2023-12-16T07:44:21.357 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283 running (100s) 72s ago 13m 475M - 19.0.0-191-g6d933094 a29c258b5d6f e5055e9fae4f 2023-12-16T07:44:21.357 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:9283 running (16m) 72s ago 16m 404M - 17.2.0 e1d6a67b021e 5219d4698403 2023-12-16T07:44:21.357 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (16m) 72s ago 16m 62.2M 2048M 17.2.0 e1d6a67b021e c0468ecbd44e 2023-12-16T07:44:21.357 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (14m) 72s ago 14m 49.3M 2048M 17.2.0 e1d6a67b021e b47a960c6102 2023-12-16T07:44:21.357 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (14m) 72s ago 14m 45.6M 2048M 17.2.0 e1d6a67b021e c659b986f201 2023-12-16T07:44:21.357 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 starting - - - - 2023-12-16T07:44:21.357 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 starting - - - - 2023-12-16T07:44:21.358 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (13m) 72s ago 13m 56.6M 2782M 17.2.0 e1d6a67b021e dbc4d9538238 2023-12-16T07:44:21.358 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (12m) 72s ago 12m 58.0M 2782M 17.2.0 e1d6a67b021e 14637e0f389c 2023-12-16T07:44:21.358 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (12m) 72s ago 12m 55.6M 2782M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:44:21.358 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (11m) 72s ago 11m 54.8M 2782M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:44:21.358 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (10m) 72s ago 10m 56.1M 3037M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:44:21.358 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (10m) 72s ago 10m 54.1M 3037M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:44:21.358 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (9m) 72s ago 9m 53.8M 3037M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:44:21.358 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (9m) 72s ago 9m 53.9M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:44:21.358 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 starting - - - - 2023-12-16T07:44:21.358 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (6m) 72s ago 6m 85.4M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:44:21.358 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (6m) 72s ago 6m 85.5M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:44:21.358 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (5m) 72s ago 5m 85.2M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:44:21.358 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (5m) 72s ago 5m 85.3M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:44:21.816 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:44:21.816 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:44:21.816 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2023-12-16T07:44:21.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:44:21.816 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:44:21.816 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2023-12-16T07:44:21.816 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 1 2023-12-16T07:44:21.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:44:21.816 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:44:21.816 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2023-12-16T07:44:21.816 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:44:21.817 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2023-12-16T07:44:21.817 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:44:21.817 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:44:21.817 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:44:21.817 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:44:21.817 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2023-12-16T07:44:21.817 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 1 2023-12-16T07:44:21.817 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:44:21.817 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:44:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:21 smithi007 bash[30436]: cephadm 2023-12-16T07:44:19.759849+0000 mgr.x (mgr.24968) 106 : cephadm [INF] Upgrade: Updating mgr.y 2023-12-16T07:44:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:21 smithi007 bash[30436]: cephadm 2023-12-16T07:44:19.780324+0000 mgr.x (mgr.24968) 107 : cephadm [INF] Deploying daemon mgr.y on smithi007 2023-12-16T07:44:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:21 smithi007 bash[30436]: cluster 2023-12-16T07:44:20.020782+0000 mgr.x (mgr.24968) 108 : cluster [DBG] pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:21 smithi007 bash[30436]: audit 2023-12-16T07:44:20.565765+0000 mgr.x (mgr.24968) 109 : audit [DBG] from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:21.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:21 smithi007 bash[25009]: cephadm 2023-12-16T07:44:19.759849+0000 mgr.x (mgr.24968) 106 : cephadm [INF] Upgrade: Updating mgr.y 2023-12-16T07:44:21.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:21 smithi007 bash[25009]: cephadm 2023-12-16T07:44:19.780324+0000 mgr.x (mgr.24968) 107 : cephadm [INF] Deploying daemon mgr.y on smithi007 2023-12-16T07:44:21.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:21 smithi007 bash[25009]: cluster 2023-12-16T07:44:20.020782+0000 mgr.x (mgr.24968) 108 : cluster [DBG] pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:21.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:21 smithi007 bash[25009]: audit 2023-12-16T07:44:20.565765+0000 mgr.x (mgr.24968) 109 : audit [DBG] from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:21.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:21 smithi169 bash[25083]: cephadm 2023-12-16T07:44:19.759849+0000 mgr.x (mgr.24968) 106 : cephadm [INF] Upgrade: Updating mgr.y 2023-12-16T07:44:21.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:21 smithi169 bash[25083]: cephadm 2023-12-16T07:44:19.780324+0000 mgr.x (mgr.24968) 107 : cephadm [INF] Deploying daemon mgr.y on smithi007 2023-12-16T07:44:21.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:21 smithi169 bash[25083]: cluster 2023-12-16T07:44:20.020782+0000 mgr.x (mgr.24968) 108 : cluster [DBG] pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:21.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:21 smithi169 bash[25083]: audit 2023-12-16T07:44:20.565765+0000 mgr.x (mgr.24968) 109 : audit [DBG] from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:22.212 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:44:22.212 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:44:22.212 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:44:22.212 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:44:22.212 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [], 2023-12-16T07:44:22.213 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "1/23 daemons upgraded", 2023-12-16T07:44:22.213 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading mgr daemons", 2023-12-16T07:44:22.213 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:44:22.213 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:44:22.561 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:22 smithi007 systemd[1]: Stopping Ceph mgr.y for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:44:22.674 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:44:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:22 smithi007 bash[25009]: audit 2023-12-16T07:44:20.938800+0000 mgr.x (mgr.24968) 110 : audit [DBG] from='client.25306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:22 smithi007 bash[25009]: audit 2023-12-16T07:44:21.331243+0000 mgr.x (mgr.24968) 111 : audit [DBG] from='client.25312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:22 smithi007 bash[25009]: audit 2023-12-16T07:44:21.813357+0000 mon.a (mon.0) 998 : audit [DBG] from='client.? 172.21.15.7:0/3989126679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:44:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:22 smithi007 bash[30436]: audit 2023-12-16T07:44:20.938800+0000 mgr.x (mgr.24968) 110 : audit [DBG] from='client.25306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:22 smithi007 bash[30436]: audit 2023-12-16T07:44:21.331243+0000 mgr.x (mgr.24968) 111 : audit [DBG] from='client.25312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:22 smithi007 bash[30436]: audit 2023-12-16T07:44:21.813357+0000 mon.a (mon.0) 998 : audit [DBG] from='client.? 172.21.15.7:0/3989126679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:44:22.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:22 smithi169 bash[25083]: audit 2023-12-16T07:44:20.938800+0000 mgr.x (mgr.24968) 110 : audit [DBG] from='client.25306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:22.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:22 smithi169 bash[25083]: audit 2023-12-16T07:44:21.331243+0000 mgr.x (mgr.24968) 111 : audit [DBG] from='client.25312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:22.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:22 smithi169 bash[25083]: audit 2023-12-16T07:44:21.813357+0000 mon.a (mon.0) 998 : audit [DBG] from='client.? 172.21.15.7:0/3989126679' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:44:23.371 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:23 smithi007 bash[61121]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mgr-y 2023-12-16T07:44:23.647 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:23 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-12-16T07:44:23.648 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:23 smithi007 bash[61194]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mgr-y 2023-12-16T07:44:23.648 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:23 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2023-12-16T07:44:23.648 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:23 smithi007 systemd[1]: Stopped Ceph mgr.y for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:44:23.648 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:23 smithi007 bash[30436]: cluster 2023-12-16T07:44:22.022472+0000 mgr.x (mgr.24968) 112 : cluster [DBG] pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:23.648 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:23 smithi007 bash[30436]: audit 2023-12-16T07:44:22.211192+0000 mgr.x (mgr.24968) 113 : audit [DBG] from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:23.648 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:23 smithi007 bash[30436]: audit 2023-12-16T07:44:22.671706+0000 mon.a (mon.0) 999 : audit [DBG] from='client.? 172.21.15.7:0/724562616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:44:23.648 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:23 smithi007 bash[25009]: cluster 2023-12-16T07:44:22.022472+0000 mgr.x (mgr.24968) 112 : cluster [DBG] pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:23.648 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:23 smithi007 bash[25009]: audit 2023-12-16T07:44:22.211192+0000 mgr.x (mgr.24968) 113 : audit [DBG] from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:23.649 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:23 smithi007 bash[25009]: audit 2023-12-16T07:44:22.671706+0000 mon.a (mon.0) 999 : audit [DBG] from='client.? 172.21.15.7:0/724562616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:44:23.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:23 smithi169 bash[25083]: cluster 2023-12-16T07:44:22.022472+0000 mgr.x (mgr.24968) 112 : cluster [DBG] pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:23.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:23 smithi169 bash[25083]: audit 2023-12-16T07:44:22.211192+0000 mgr.x (mgr.24968) 113 : audit [DBG] from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:23.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:23 smithi169 bash[25083]: audit 2023-12-16T07:44:22.671706+0000 mon.a (mon.0) 999 : audit [DBG] from='client.? 172.21.15.7:0/724562616' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:44:24.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:23 smithi007 systemd[1]: Started Ceph mgr.y for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:44:24.952 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:24 smithi007 bash[61260]: debug 2023-12-16T07:44:24.781+0000 7f7717982700 1 -- 172.21.15.7:0/3896293830 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563b3abb5520 con 0x563b3b932400 2023-12-16T07:44:24.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:24 smithi007 bash[30436]: audit 2023-12-16T07:44:23.696478+0000 mon.a (mon.0) 1000 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:24.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:24 smithi007 bash[30436]: audit 2023-12-16T07:44:23.706535+0000 mon.a (mon.0) 1001 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:24.952 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:24 smithi007 bash[30436]: audit 2023-12-16T07:44:24.170274+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:24.952 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:24 smithi007 bash[25009]: audit 2023-12-16T07:44:23.696478+0000 mon.a (mon.0) 1000 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:24.952 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:24 smithi007 bash[25009]: audit 2023-12-16T07:44:23.706535+0000 mon.a (mon.0) 1001 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:24.952 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:24 smithi007 bash[25009]: audit 2023-12-16T07:44:24.170274+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:25.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:24 smithi169 bash[25083]: audit 2023-12-16T07:44:23.696478+0000 mon.a (mon.0) 1000 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:25.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:24 smithi169 bash[25083]: audit 2023-12-16T07:44:23.706535+0000 mon.a (mon.0) 1001 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:25.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:24 smithi169 bash[25083]: audit 2023-12-16T07:44:24.170274+0000 mon.b (mon.2) 80 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:25.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:24 smithi007 bash[61260]: debug 2023-12-16T07:44:24.945+0000 7f7721b29200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T07:44:26.000 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:25 smithi007 bash[61260]: debug 2023-12-16T07:44:25.889+0000 7f7721b29200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-16T07:44:26.001 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:25 smithi007 bash[30436]: cluster 2023-12-16T07:44:24.023222+0000 mgr.x (mgr.24968) 114 : cluster [DBG] pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:26.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:25 smithi007 bash[25009]: cluster 2023-12-16T07:44:24.023222+0000 mgr.x (mgr.24968) 114 : cluster [DBG] pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:26.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:25 smithi169 bash[25083]: cluster 2023-12-16T07:44:24.023222+0000 mgr.x (mgr.24968) 114 : cluster [DBG] pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:26.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:25 smithi007 bash[61260]: debug 2023-12-16T07:44:25.997+0000 7f7721b29200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T07:44:26.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:26 smithi007 bash[61260]: debug 2023-12-16T07:44:26.101+0000 7f7721b29200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T07:44:26.759 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:26 smithi007 bash[61260]: debug 2023-12-16T07:44:26.377+0000 7f7721b29200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:44:27.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:26 smithi007 bash[61260]: debug 2023-12-16T07:44:26.753+0000 7f7721b29200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T07:44:27.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:26 smithi007 bash[61260]: debug 2023-12-16T07:44:26.917+0000 7f7721b29200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T07:44:27.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:27 smithi007 bash[61260]: debug 2023-12-16T07:44:27.137+0000 7f7721b29200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T07:44:28.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:27 smithi007 bash[25009]: cluster 2023-12-16T07:44:26.024916+0000 mgr.x (mgr.24968) 115 : cluster [DBG] pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:28.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:27 smithi007 bash[30436]: cluster 2023-12-16T07:44:26.024916+0000 mgr.x (mgr.24968) 115 : cluster [DBG] pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:28.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:27 smithi169 bash[25083]: cluster 2023-12-16T07:44:26.024916+0000 mgr.x (mgr.24968) 115 : cluster [DBG] pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:28.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:28 smithi007 bash[61260]: debug 2023-12-16T07:44:28.181+0000 7f7721b29200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T07:44:28.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:28 smithi007 bash[61260]: debug 2023-12-16T07:44:28.297+0000 7f7721b29200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T07:44:28.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:28 smithi007 bash[61260]: debug 2023-12-16T07:44:28.417+0000 7f7721b29200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T07:44:28.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:28 smithi169 bash[45188]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:44:28] "GET /metrics HTTP/1.1" 200 38126 "" "Prometheus/2.33.4" 2023-12-16T07:44:28.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:44:28 smithi169 bash[47208]: ts=2023-12-16T07:44:28.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:44:29.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:28 smithi007 bash[61260]: debug 2023-12-16T07:44:28.909+0000 7f7721b29200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T07:44:29.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:29 smithi007 bash[61260]: debug 2023-12-16T07:44:29.149+0000 7f7721b29200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T07:44:29.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:29 smithi007 bash[61260]: debug 2023-12-16T07:44:29.637+0000 7f7721b29200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T07:44:29.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:29 smithi007 bash[30436]: cluster 2023-12-16T07:44:28.025696+0000 mgr.x (mgr.24968) 116 : cluster [DBG] pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:29.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:29 smithi007 bash[25009]: cluster 2023-12-16T07:44:28.025696+0000 mgr.x (mgr.24968) 116 : cluster [DBG] pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:30.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:29 smithi169 bash[25083]: cluster 2023-12-16T07:44:28.025696+0000 mgr.x (mgr.24968) 116 : cluster [DBG] pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:30.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:30 smithi007 bash[61260]: debug 2023-12-16T07:44:30.177+0000 7f7721b29200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:44:31.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:30 smithi007 bash[61260]: debug 2023-12-16T07:44:30.765+0000 7f7721b29200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T07:44:31.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:30 smithi007 bash[61260]: debug 2023-12-16T07:44:30.881+0000 7f7721b29200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T07:44:31.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:30 smithi007 bash[30436]: audit 2023-12-16T07:44:29.556951+0000 mgr.x (mgr.24968) 117 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:31.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:30 smithi007 bash[30436]: audit 2023-12-16T07:44:30.204304+0000 mon.a (mon.0) 1002 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:30 smithi007 bash[30436]: audit 2023-12-16T07:44:30.211297+0000 mon.a (mon.0) 1003 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:30 smithi007 bash[30436]: audit 2023-12-16T07:44:30.220232+0000 mon.a (mon.0) 1004 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:30 smithi007 bash[30436]: audit 2023-12-16T07:44:30.228398+0000 mon.a (mon.0) 1005 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:30 smithi007 bash[25009]: audit 2023-12-16T07:44:29.556951+0000 mgr.x (mgr.24968) 117 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:31.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:30 smithi007 bash[25009]: audit 2023-12-16T07:44:30.204304+0000 mon.a (mon.0) 1002 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:30 smithi007 bash[25009]: audit 2023-12-16T07:44:30.211297+0000 mon.a (mon.0) 1003 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:30 smithi007 bash[25009]: audit 2023-12-16T07:44:30.220232+0000 mon.a (mon.0) 1004 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:30 smithi007 bash[25009]: audit 2023-12-16T07:44:30.228398+0000 mon.a (mon.0) 1005 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:30 smithi169 bash[25083]: audit 2023-12-16T07:44:29.556951+0000 mgr.x (mgr.24968) 117 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:31.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:30 smithi169 bash[25083]: audit 2023-12-16T07:44:30.204304+0000 mon.a (mon.0) 1002 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:30 smithi169 bash[25083]: audit 2023-12-16T07:44:30.211297+0000 mon.a (mon.0) 1003 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:30 smithi169 bash[25083]: audit 2023-12-16T07:44:30.220232+0000 mon.a (mon.0) 1004 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:30 smithi169 bash[25083]: audit 2023-12-16T07:44:30.228398+0000 mon.a (mon.0) 1005 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.513 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:31 smithi007 bash[61260]: debug 2023-12-16T07:44:31.241+0000 7f7721b29200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T07:44:31.513 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:31 smithi007 bash[61260]: debug 2023-12-16T07:44:31.357+0000 7f7721b29200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T07:44:31.768 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:31 smithi007 bash[61260]: debug 2023-12-16T07:44:31.509+0000 7f7721b29200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T07:44:31.768 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:31 smithi007 bash[61260]: debug 2023-12-16T07:44:31.629+0000 7f7721b29200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T07:44:31.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:31 smithi007 bash[30436]: cluster 2023-12-16T07:44:30.026481+0000 mgr.x (mgr.24968) 118 : cluster [DBG] pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:31.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:31 smithi007 bash[30436]: audit 2023-12-16T07:44:31.326920+0000 mon.a (mon.0) 1006 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:31 smithi007 bash[30436]: audit 2023-12-16T07:44:31.337836+0000 mon.a (mon.0) 1007 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.769 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:31 smithi007 bash[25009]: cluster 2023-12-16T07:44:30.026481+0000 mgr.x (mgr.24968) 118 : cluster [DBG] pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:31.769 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:31 smithi007 bash[25009]: audit 2023-12-16T07:44:31.326920+0000 mon.a (mon.0) 1006 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:31.770 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:31 smithi007 bash[25009]: audit 2023-12-16T07:44:31.337836+0000 mon.a (mon.0) 1007 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:32.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:31 smithi007 bash[61260]: debug 2023-12-16T07:44:31.761+0000 7f7721b29200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T07:44:32.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:31 smithi007 bash[61260]: debug 2023-12-16T07:44:31.957+0000 7f7721b29200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T07:44:32.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:31 smithi169 bash[25083]: cluster 2023-12-16T07:44:30.026481+0000 mgr.x (mgr.24968) 118 : cluster [DBG] pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:32.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:31 smithi169 bash[25083]: audit 2023-12-16T07:44:31.326920+0000 mon.a (mon.0) 1006 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:32.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:31 smithi169 bash[25083]: audit 2023-12-16T07:44:31.337836+0000 mon.a (mon.0) 1007 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:32.573 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:32 smithi007 bash[61260]: debug 2023-12-16T07:44:32.089+0000 7f7721b29200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T07:44:33.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:33 smithi007 bash[30436]: cluster 2023-12-16T07:44:32.028062+0000 mgr.x (mgr.24968) 119 : cluster [DBG] pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:33.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:33 smithi007 bash[25009]: cluster 2023-12-16T07:44:32.028062+0000 mgr.x (mgr.24968) 119 : cluster [DBG] pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:33.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:33 smithi007 bash[61260]: debug 2023-12-16T07:44:33.365+0000 7f7721b29200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T07:44:33.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:33 smithi007 bash[61260]: [16/Dec/2023:07:44:33] ENGINE Bus STARTING 2023-12-16T07:44:33.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:33 smithi007 bash[61260]: CherryPy Checker: 2023-12-16T07:44:33.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:33 smithi007 bash[61260]: The Application mounted at '' has an empty config. 2023-12-16T07:44:33.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:33 smithi007 bash[61260]: [16/Dec/2023:07:44:33] ENGINE Serving on http://:::9283 2023-12-16T07:44:33.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:33 smithi007 bash[61260]: [16/Dec/2023:07:44:33] ENGINE Bus STARTED 2023-12-16T07:44:33.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:33 smithi169 bash[25083]: cluster 2023-12-16T07:44:32.028062+0000 mgr.x (mgr.24968) 119 : cluster [DBG] pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:34.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:34 smithi007 bash[25009]: cluster 2023-12-16T07:44:33.393647+0000 mon.a (mon.0) 1008 : cluster [DBG] Standby manager daemon y restarted 2023-12-16T07:44:34.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:34 smithi007 bash[25009]: cluster 2023-12-16T07:44:33.393818+0000 mon.a (mon.0) 1009 : cluster [DBG] Standby manager daemon y started 2023-12-16T07:44:34.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:34 smithi007 bash[25009]: audit 2023-12-16T07:44:33.399069+0000 mon.a (mon.0) 1010 : audit [DBG] from='mgr.? 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-12-16T07:44:34.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:34 smithi007 bash[25009]: audit 2023-12-16T07:44:33.400010+0000 mon.a (mon.0) 1011 : audit [DBG] from='mgr.? 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:44:34.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:34 smithi007 bash[25009]: audit 2023-12-16T07:44:33.405502+0000 mon.a (mon.0) 1012 : audit [DBG] from='mgr.? 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-12-16T07:44:34.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:34 smithi007 bash[25009]: audit 2023-12-16T07:44:33.408518+0000 mon.a (mon.0) 1013 : audit [DBG] from='mgr.? 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:44:34.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:34 smithi007 bash[30436]: cluster 2023-12-16T07:44:33.393647+0000 mon.a (mon.0) 1008 : cluster [DBG] Standby manager daemon y restarted 2023-12-16T07:44:34.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:34 smithi007 bash[30436]: cluster 2023-12-16T07:44:33.393818+0000 mon.a (mon.0) 1009 : cluster [DBG] Standby manager daemon y started 2023-12-16T07:44:34.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:34 smithi007 bash[30436]: audit 2023-12-16T07:44:33.399069+0000 mon.a (mon.0) 1010 : audit [DBG] from='mgr.? 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-12-16T07:44:34.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:34 smithi007 bash[30436]: audit 2023-12-16T07:44:33.400010+0000 mon.a (mon.0) 1011 : audit [DBG] from='mgr.? 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:44:34.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:34 smithi007 bash[30436]: audit 2023-12-16T07:44:33.405502+0000 mon.a (mon.0) 1012 : audit [DBG] from='mgr.? 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-12-16T07:44:34.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:34 smithi007 bash[30436]: audit 2023-12-16T07:44:33.408518+0000 mon.a (mon.0) 1013 : audit [DBG] from='mgr.? 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:44:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:34 smithi169 bash[25083]: cluster 2023-12-16T07:44:33.393647+0000 mon.a (mon.0) 1008 : cluster [DBG] Standby manager daemon y restarted 2023-12-16T07:44:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:34 smithi169 bash[25083]: cluster 2023-12-16T07:44:33.393818+0000 mon.a (mon.0) 1009 : cluster [DBG] Standby manager daemon y started 2023-12-16T07:44:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:34 smithi169 bash[25083]: audit 2023-12-16T07:44:33.399069+0000 mon.a (mon.0) 1010 : audit [DBG] from='mgr.? 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-12-16T07:44:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:34 smithi169 bash[25083]: audit 2023-12-16T07:44:33.400010+0000 mon.a (mon.0) 1011 : audit [DBG] from='mgr.? 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:44:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:34 smithi169 bash[25083]: audit 2023-12-16T07:44:33.405502+0000 mon.a (mon.0) 1012 : audit [DBG] from='mgr.? 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-12-16T07:44:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:34 smithi169 bash[25083]: audit 2023-12-16T07:44:33.408518+0000 mon.a (mon.0) 1013 : audit [DBG] from='mgr.? 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:44:35.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:35 smithi007 bash[25009]: cluster 2023-12-16T07:44:34.028860+0000 mgr.x (mgr.24968) 120 : cluster [DBG] pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:35.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:35 smithi007 bash[25009]: cluster 2023-12-16T07:44:34.282104+0000 mon.a (mon.0) 1014 : cluster [DBG] mgrmap e28: x(active, since 101s), standbys: y 2023-12-16T07:44:35.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:35 smithi007 bash[30436]: cluster 2023-12-16T07:44:34.028860+0000 mgr.x (mgr.24968) 120 : cluster [DBG] pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:35.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:35 smithi007 bash[30436]: cluster 2023-12-16T07:44:34.282104+0000 mon.a (mon.0) 1014 : cluster [DBG] mgrmap e28: x(active, since 101s), standbys: y 2023-12-16T07:44:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:35 smithi169 bash[25083]: cluster 2023-12-16T07:44:34.028860+0000 mgr.x (mgr.24968) 120 : cluster [DBG] pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:35 smithi169 bash[25083]: cluster 2023-12-16T07:44:34.282104+0000 mon.a (mon.0) 1014 : cluster [DBG] mgrmap e28: x(active, since 101s), standbys: y 2023-12-16T07:44:37.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:37 smithi007 bash[30436]: cluster 2023-12-16T07:44:36.030611+0000 mgr.x (mgr.24968) 121 : cluster [DBG] pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:37.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:37 smithi007 bash[25009]: cluster 2023-12-16T07:44:36.030611+0000 mgr.x (mgr.24968) 121 : cluster [DBG] pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:37.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:37 smithi169 bash[25083]: cluster 2023-12-16T07:44:36.030611+0000 mgr.x (mgr.24968) 121 : cluster [DBG] pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:44:38.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:38 smithi169 bash[45188]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:44:38] "GET /metrics HTTP/1.1" 200 38126 "" "Prometheus/2.33.4" 2023-12-16T07:44:38.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:44:38 smithi169 bash[47208]: ts=2023-12-16T07:44:38.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:44:39.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:39 smithi007 bash[30436]: cluster 2023-12-16T07:44:38.031376+0000 mgr.x (mgr.24968) 122 : cluster [DBG] pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:39.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:39 smithi007 bash[30436]: audit 2023-12-16T07:44:39.170701+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:39.538 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:39 smithi007 bash[25009]: cluster 2023-12-16T07:44:38.031376+0000 mgr.x (mgr.24968) 122 : cluster [DBG] pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:39.538 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:39 smithi007 bash[25009]: audit 2023-12-16T07:44:39.170701+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:39.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:39 smithi169 bash[25083]: cluster 2023-12-16T07:44:38.031376+0000 mgr.x (mgr.24968) 122 : cluster [DBG] pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:39.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:39 smithi169 bash[25083]: audit 2023-12-16T07:44:39.170701+0000 mon.b (mon.2) 81 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:40.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:40 smithi007 bash[30436]: audit 2023-12-16T07:44:39.565108+0000 mgr.x (mgr.24968) 123 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:40.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:40 smithi007 bash[25009]: audit 2023-12-16T07:44:39.565108+0000 mgr.x (mgr.24968) 123 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:40.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:40 smithi169 bash[25083]: audit 2023-12-16T07:44:39.565108+0000 mgr.x (mgr.24968) 123 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:41.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:41 smithi007 bash[30436]: cluster 2023-12-16T07:44:40.032775+0000 mgr.x (mgr.24968) 124 : cluster [DBG] pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:41.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:41 smithi007 bash[25009]: cluster 2023-12-16T07:44:40.032775+0000 mgr.x (mgr.24968) 124 : cluster [DBG] pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:41.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:41 smithi169 bash[25083]: cluster 2023-12-16T07:44:40.032775+0000 mgr.x (mgr.24968) 124 : cluster [DBG] pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:44:42.905 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:42 smithi169 bash[45188]: debug 2023-12-16T07:44:42.682+0000 7fd32f02d700 -1 mgr handle_mgr_map I was active but no longer am 2023-12-16T07:44:42.905 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:42 smithi169 bash[45188]: ignoring --setuser ceph since I am not root 2023-12-16T07:44:42.905 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:42 smithi169 bash[45188]: ignoring --setgroup ceph since I am not root 2023-12-16T07:44:42.905 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:42 smithi169 bash[45188]: debug 2023-12-16T07:44:42.738+0000 7fa8d08a6700 1 -- 172.21.15.169:0/314547901 <== mon.2 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a8a0a39520 con 0x55a8a17b4400 2023-12-16T07:44:42.906 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:42 smithi169 bash[25083]: audit 2023-12-16T07:44:41.647586+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:42.906 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:42 smithi169 bash[25083]: audit 2023-12-16T07:44:41.661305+0000 mon.a (mon.0) 1016 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:42.906 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:42 smithi169 bash[25083]: audit 2023-12-16T07:44:41.666343+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:44:42.906 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:42 smithi169 bash[25083]: audit 2023-12-16T07:44:41.667956+0000 mon.b (mon.2) 83 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:44:42.906 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:42 smithi169 bash[25083]: audit 2023-12-16T07:44:41.677103+0000 mon.a (mon.0) 1017 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:42.906 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:42 smithi169 bash[25083]: audit 2023-12-16T07:44:41.767479+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:44:42.906 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:42 smithi169 bash[25083]: audit 2023-12-16T07:44:41.772793+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-12-16T07:44:42.906 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:42 smithi169 bash[25083]: audit 2023-12-16T07:44:41.773734+0000 mon.b (mon.2) 85 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-12-16T07:44:42.906 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:42 smithi169 bash[25083]: cluster 2023-12-16T07:44:41.782840+0000 mon.a (mon.0) 1019 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2023-12-16T07:44:42.927 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[61260]: [16/Dec/2023:07:44:42] ENGINE Bus STOPPING 2023-12-16T07:44:42.927 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[61260]: [16/Dec/2023:07:44:42] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2023-12-16T07:44:42.927 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[61260]: [16/Dec/2023:07:44:42] ENGINE Bus STOPPED 2023-12-16T07:44:42.927 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[61260]: [16/Dec/2023:07:44:42] ENGINE Bus STARTING 2023-12-16T07:44:42.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[30436]: audit 2023-12-16T07:44:41.647586+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:42.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[30436]: audit 2023-12-16T07:44:41.661305+0000 mon.a (mon.0) 1016 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:42.927 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[30436]: audit 2023-12-16T07:44:41.666343+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:44:42.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[30436]: audit 2023-12-16T07:44:41.667956+0000 mon.b (mon.2) 83 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:44:42.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[30436]: audit 2023-12-16T07:44:41.677103+0000 mon.a (mon.0) 1017 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:42.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[30436]: audit 2023-12-16T07:44:41.767479+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:44:42.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[30436]: audit 2023-12-16T07:44:41.772793+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-12-16T07:44:42.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[30436]: audit 2023-12-16T07:44:41.773734+0000 mon.b (mon.2) 85 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-12-16T07:44:42.928 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[30436]: cluster 2023-12-16T07:44:41.782840+0000 mon.a (mon.0) 1019 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2023-12-16T07:44:42.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[25009]: audit 2023-12-16T07:44:41.647586+0000 mon.a (mon.0) 1015 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:42.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[25009]: audit 2023-12-16T07:44:41.661305+0000 mon.a (mon.0) 1016 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:42.928 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[25009]: audit 2023-12-16T07:44:41.666343+0000 mon.b (mon.2) 82 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:44:42.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[25009]: audit 2023-12-16T07:44:41.667956+0000 mon.b (mon.2) 83 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:44:42.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[25009]: audit 2023-12-16T07:44:41.677103+0000 mon.a (mon.0) 1017 : audit [INF] from='mgr.24968 ' entity='mgr.x' 2023-12-16T07:44:42.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[25009]: audit 2023-12-16T07:44:41.767479+0000 mon.b (mon.2) 84 : audit [DBG] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:44:42.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[25009]: audit 2023-12-16T07:44:41.772793+0000 mon.a (mon.0) 1018 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-12-16T07:44:42.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[25009]: audit 2023-12-16T07:44:41.773734+0000 mon.b (mon.2) 85 : audit [INF] from='mgr.24968 172.21.15.169:0/1968538384' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2023-12-16T07:44:42.929 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[25009]: cluster 2023-12-16T07:44:41.782840+0000 mon.a (mon.0) 1019 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2023-12-16T07:44:43.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:42 smithi169 bash[45188]: debug 2023-12-16T07:44:42.898+0000 7fa8da62f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T07:44:43.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[61260]: [16/Dec/2023:07:44:42] ENGINE Serving on http://:::9283 2023-12-16T07:44:43.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:44:42 smithi007 bash[61260]: [16/Dec/2023:07:44:42] ENGINE Bus STARTED 2023-12-16T07:44:43.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[45188]: debug 2023-12-16T07:44:43.834+0000 7fa8da62f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-16T07:44:43.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.685452+0000 mon.a (mon.0) 1020 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2023-12-16T07:44:43.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: cluster 2023-12-16T07:44:42.685550+0000 mon.a (mon.0) 1021 : cluster [DBG] mgrmap e29: y(active, starting, since 0.911361s) 2023-12-16T07:44:43.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.687640+0000 mon.a (mon.0) 1022 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:44:43.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.687948+0000 mon.a (mon.0) 1023 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:44:43.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.688141+0000 mon.a (mon.0) 1024 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:44:43.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.688591+0000 mon.a (mon.0) 1025 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:44:43.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.688912+0000 mon.a (mon.0) 1026 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:44:43.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.689207+0000 mon.a (mon.0) 1027 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:44:43.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.689485+0000 mon.a (mon.0) 1028 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:44:43.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.689847+0000 mon.a (mon.0) 1029 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:44:43.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.690148+0000 mon.a (mon.0) 1030 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:44:43.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.690418+0000 mon.a (mon.0) 1031 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:44:43.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.690683+0000 mon.a (mon.0) 1032 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:44:43.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.690942+0000 mon.a (mon.0) 1033 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:44:43.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.691214+0000 mon.a (mon.0) 1034 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:44:43.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.691422+0000 mon.a (mon.0) 1035 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:44:43.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.691946+0000 mon.a (mon.0) 1036 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:44:43.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: cluster 2023-12-16T07:44:42.734668+0000 mon.a (mon.0) 1037 : cluster [INF] Manager daemon y is now available 2023-12-16T07:44:43.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.795325+0000 mon.a (mon.0) 1038 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:44:43.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.852190+0000 mon.a (mon.0) 1039 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:43.936 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.869021+0000 mon.a (mon.0) 1040 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:44:43.936 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[25083]: audit 2023-12-16T07:44:42.923354+0000 mon.a (mon.0) 1041 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:44:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.685452+0000 mon.a (mon.0) 1020 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2023-12-16T07:44:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: cluster 2023-12-16T07:44:42.685550+0000 mon.a (mon.0) 1021 : cluster [DBG] mgrmap e29: y(active, starting, since 0.911361s) 2023-12-16T07:44:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.687640+0000 mon.a (mon.0) 1022 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:44:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.687948+0000 mon.a (mon.0) 1023 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:44:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.688141+0000 mon.a (mon.0) 1024 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:44:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.688591+0000 mon.a (mon.0) 1025 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:44:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.688912+0000 mon.a (mon.0) 1026 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:44:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.689207+0000 mon.a (mon.0) 1027 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:44:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.689485+0000 mon.a (mon.0) 1028 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:44:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.689847+0000 mon.a (mon.0) 1029 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:44:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.690148+0000 mon.a (mon.0) 1030 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:44:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.690418+0000 mon.a (mon.0) 1031 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:44:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.690683+0000 mon.a (mon.0) 1032 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:44:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.690942+0000 mon.a (mon.0) 1033 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:44:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.691214+0000 mon.a (mon.0) 1034 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:44:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.691422+0000 mon.a (mon.0) 1035 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:44:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.691946+0000 mon.a (mon.0) 1036 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:44:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: cluster 2023-12-16T07:44:42.734668+0000 mon.a (mon.0) 1037 : cluster [INF] Manager daemon y is now available 2023-12-16T07:44:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.795325+0000 mon.a (mon.0) 1038 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:44:44.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.685452+0000 mon.a (mon.0) 1020 : audit [INF] from='mgr.24968 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2023-12-16T07:44:44.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: cluster 2023-12-16T07:44:42.685550+0000 mon.a (mon.0) 1021 : cluster [DBG] mgrmap e29: y(active, starting, since 0.911361s) 2023-12-16T07:44:44.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.687640+0000 mon.a (mon.0) 1022 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:44:44.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.687948+0000 mon.a (mon.0) 1023 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:44:44.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.688141+0000 mon.a (mon.0) 1024 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:44:44.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.688591+0000 mon.a (mon.0) 1025 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:44:44.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.688912+0000 mon.a (mon.0) 1026 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:44:44.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.689207+0000 mon.a (mon.0) 1027 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:44:44.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.689485+0000 mon.a (mon.0) 1028 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:44:44.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.689847+0000 mon.a (mon.0) 1029 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:44:44.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.690148+0000 mon.a (mon.0) 1030 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:44:44.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.690418+0000 mon.a (mon.0) 1031 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:44:44.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.690683+0000 mon.a (mon.0) 1032 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:44:44.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.690942+0000 mon.a (mon.0) 1033 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:44:44.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.691214+0000 mon.a (mon.0) 1034 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:44:44.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.691422+0000 mon.a (mon.0) 1035 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:44:44.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.691946+0000 mon.a (mon.0) 1036 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:44:44.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: cluster 2023-12-16T07:44:42.734668+0000 mon.a (mon.0) 1037 : cluster [INF] Manager daemon y is now available 2023-12-16T07:44:44.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.795325+0000 mon.a (mon.0) 1038 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:44:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.852190+0000 mon.a (mon.0) 1039 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.869021+0000 mon.a (mon.0) 1040 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:44:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[25009]: audit 2023-12-16T07:44:42.923354+0000 mon.a (mon.0) 1041 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:44:44.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.852190+0000 mon.a (mon.0) 1039 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:44.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.869021+0000 mon.a (mon.0) 1040 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:44:44.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:43 smithi007 bash[30436]: audit 2023-12-16T07:44:42.923354+0000 mon.a (mon.0) 1041 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:44:44.302 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:43 smithi169 bash[45188]: debug 2023-12-16T07:44:43.934+0000 7fa8da62f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T07:44:44.302 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:44 smithi169 bash[45188]: debug 2023-12-16T07:44:44.034+0000 7fa8da62f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T07:44:44.657 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:44 smithi169 bash[45188]: debug 2023-12-16T07:44:44.294+0000 7fa8da62f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:44:44.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:44 smithi169 bash[45188]: debug 2023-12-16T07:44:44.650+0000 7fa8da62f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T07:44:44.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:44 smithi169 bash[45188]: debug 2023-12-16T07:44:44.810+0000 7fa8da62f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T07:44:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:44 smithi169 bash[25083]: cephadm 2023-12-16T07:44:43.493121+0000 mgr.y (mgr.15429) 1 : cephadm [INF] [16/Dec/2023:07:44:43] ENGINE Bus STARTING 2023-12-16T07:44:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:44 smithi169 bash[25083]: cephadm 2023-12-16T07:44:43.595572+0000 mgr.y (mgr.15429) 2 : cephadm [INF] [16/Dec/2023:07:44:43] ENGINE Serving on http://172.21.15.7:8765 2023-12-16T07:44:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:44 smithi169 bash[25083]: cluster 2023-12-16T07:44:43.694032+0000 mon.a (mon.0) 1042 : cluster [DBG] mgrmap e30: y(active, since 1.91984s) 2023-12-16T07:44:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:44 smithi169 bash[25083]: cephadm 2023-12-16T07:44:43.731979+0000 mgr.y (mgr.15429) 3 : cephadm [INF] [16/Dec/2023:07:44:43] ENGINE Serving on https://172.21.15.7:7150 2023-12-16T07:44:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:44 smithi169 bash[25083]: cephadm 2023-12-16T07:44:43.732658+0000 mgr.y (mgr.15429) 4 : cephadm [INF] [16/Dec/2023:07:44:43] ENGINE Bus STARTED 2023-12-16T07:44:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:44 smithi169 bash[25083]: cluster 2023-12-16T07:44:43.746501+0000 mgr.y (mgr.15429) 5 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:44:45.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:44 smithi007 bash[25009]: cephadm 2023-12-16T07:44:43.493121+0000 mgr.y (mgr.15429) 1 : cephadm [INF] [16/Dec/2023:07:44:43] ENGINE Bus STARTING 2023-12-16T07:44:45.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:44 smithi007 bash[25009]: cephadm 2023-12-16T07:44:43.595572+0000 mgr.y (mgr.15429) 2 : cephadm [INF] [16/Dec/2023:07:44:43] ENGINE Serving on http://172.21.15.7:8765 2023-12-16T07:44:45.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:44 smithi007 bash[25009]: cluster 2023-12-16T07:44:43.694032+0000 mon.a (mon.0) 1042 : cluster [DBG] mgrmap e30: y(active, since 1.91984s) 2023-12-16T07:44:45.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:44 smithi007 bash[25009]: cephadm 2023-12-16T07:44:43.731979+0000 mgr.y (mgr.15429) 3 : cephadm [INF] [16/Dec/2023:07:44:43] ENGINE Serving on https://172.21.15.7:7150 2023-12-16T07:44:45.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:44 smithi007 bash[25009]: cephadm 2023-12-16T07:44:43.732658+0000 mgr.y (mgr.15429) 4 : cephadm [INF] [16/Dec/2023:07:44:43] ENGINE Bus STARTED 2023-12-16T07:44:45.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:44 smithi007 bash[25009]: cluster 2023-12-16T07:44:43.746501+0000 mgr.y (mgr.15429) 5 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:44:45.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:44 smithi007 bash[30436]: cephadm 2023-12-16T07:44:43.493121+0000 mgr.y (mgr.15429) 1 : cephadm [INF] [16/Dec/2023:07:44:43] ENGINE Bus STARTING 2023-12-16T07:44:45.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:44 smithi007 bash[30436]: cephadm 2023-12-16T07:44:43.595572+0000 mgr.y (mgr.15429) 2 : cephadm [INF] [16/Dec/2023:07:44:43] ENGINE Serving on http://172.21.15.7:8765 2023-12-16T07:44:45.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:44 smithi007 bash[30436]: cluster 2023-12-16T07:44:43.694032+0000 mon.a (mon.0) 1042 : cluster [DBG] mgrmap e30: y(active, since 1.91984s) 2023-12-16T07:44:45.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:44 smithi007 bash[30436]: cephadm 2023-12-16T07:44:43.731979+0000 mgr.y (mgr.15429) 3 : cephadm [INF] [16/Dec/2023:07:44:43] ENGINE Serving on https://172.21.15.7:7150 2023-12-16T07:44:45.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:44 smithi007 bash[30436]: cephadm 2023-12-16T07:44:43.732658+0000 mgr.y (mgr.15429) 4 : cephadm [INF] [16/Dec/2023:07:44:43] ENGINE Bus STARTED 2023-12-16T07:44:45.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:44 smithi007 bash[30436]: cluster 2023-12-16T07:44:43.746501+0000 mgr.y (mgr.15429) 5 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:44:45.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:45 smithi169 bash[45188]: debug 2023-12-16T07:44:45.042+0000 7fa8da62f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T07:44:46.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:45 smithi007 bash[25009]: cluster 2023-12-16T07:44:44.690643+0000 mgr.y (mgr.15429) 6 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:44:46.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:45 smithi007 bash[30436]: cluster 2023-12-16T07:44:44.690643+0000 mgr.y (mgr.15429) 6 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:44:46.093 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:45 smithi169 bash[25083]: cluster 2023-12-16T07:44:44.690643+0000 mgr.y (mgr.15429) 6 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:44:46.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:46 smithi169 bash[45188]: debug 2023-12-16T07:44:46.086+0000 7fa8da62f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T07:44:46.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:46 smithi169 bash[45188]: debug 2023-12-16T07:44:46.198+0000 7fa8da62f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T07:44:46.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:46 smithi169 bash[45188]: debug 2023-12-16T07:44:46.318+0000 7fa8da62f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T07:44:47.017 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:46 smithi169 bash[25083]: cluster 2023-12-16T07:44:45.712278+0000 mon.a (mon.0) 1043 : cluster [DBG] mgrmap e31: y(active, since 3s) 2023-12-16T07:44:47.017 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:46 smithi169 bash[45188]: debug 2023-12-16T07:44:46.790+0000 7fa8da62f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T07:44:47.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:46 smithi007 bash[25009]: cluster 2023-12-16T07:44:45.712278+0000 mon.a (mon.0) 1043 : cluster [DBG] mgrmap e31: y(active, since 3s) 2023-12-16T07:44:47.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:46 smithi007 bash[30436]: cluster 2023-12-16T07:44:45.712278+0000 mon.a (mon.0) 1043 : cluster [DBG] mgrmap e31: y(active, since 3s) 2023-12-16T07:44:47.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:47 smithi169 bash[45188]: debug 2023-12-16T07:44:47.010+0000 7fa8da62f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T07:44:47.730 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:47 smithi169 bash[45188]: debug 2023-12-16T07:44:47.458+0000 7fa8da62f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T07:44:48.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:47 smithi007 bash[25009]: cluster 2023-12-16T07:44:46.691444+0000 mgr.y (mgr.15429) 7 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:44:48.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:47 smithi007 bash[30436]: cluster 2023-12-16T07:44:46.691444+0000 mgr.y (mgr.15429) 7 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:44:48.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:47 smithi169 bash[25083]: cluster 2023-12-16T07:44:46.691444+0000 mgr.y (mgr.15429) 7 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:44:48.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:47 smithi169 bash[45188]: debug 2023-12-16T07:44:47.954+0000 7fa8da62f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:44:48.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:48 smithi169 bash[45188]: debug 2023-12-16T07:44:48.538+0000 7fa8da62f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T07:44:48.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:48 smithi169 bash[45188]: debug 2023-12-16T07:44:48.666+0000 7fa8da62f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T07:44:48.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:44:48 smithi169 bash[47208]: ts=2023-12-16T07:44:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:44:49.354 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:49 smithi169 bash[45188]: debug 2023-12-16T07:44:49.046+0000 7fa8da62f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T07:44:49.354 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:49 smithi169 bash[45188]: debug 2023-12-16T07:44:49.178+0000 7fa8da62f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T07:44:49.619 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:49 smithi169 bash[45188]: debug 2023-12-16T07:44:49.346+0000 7fa8da62f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T07:44:49.619 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:49 smithi169 bash[45188]: debug 2023-12-16T07:44:49.486+0000 7fa8da62f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T07:44:49.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:49 smithi169 bash[25083]: cluster 2023-12-16T07:44:48.692225+0000 mgr.y (mgr.15429) 8 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:44:49.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:49 smithi169 bash[25083]: audit 2023-12-16T07:44:49.585341+0000 mon.a (mon.0) 1044 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:49.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:49 smithi169 bash[25083]: audit 2023-12-16T07:44:49.595487+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:49.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:49 smithi169 bash[45188]: debug 2023-12-16T07:44:49.614+0000 7fa8da62f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T07:44:49.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:49 smithi169 bash[45188]: debug 2023-12-16T07:44:49.810+0000 7fa8da62f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T07:44:50.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:49 smithi007 bash[30436]: cluster 2023-12-16T07:44:48.692225+0000 mgr.y (mgr.15429) 8 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:44:50.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:49 smithi007 bash[30436]: audit 2023-12-16T07:44:49.585341+0000 mon.a (mon.0) 1044 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:50.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:49 smithi007 bash[30436]: audit 2023-12-16T07:44:49.595487+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:50.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:49 smithi007 bash[25009]: cluster 2023-12-16T07:44:48.692225+0000 mgr.y (mgr.15429) 8 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:44:50.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:49 smithi007 bash[25009]: audit 2023-12-16T07:44:49.585341+0000 mon.a (mon.0) 1044 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:50.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:49 smithi007 bash[25009]: audit 2023-12-16T07:44:49.595487+0000 mon.a (mon.0) 1045 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:50.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:49 smithi169 bash[45188]: debug 2023-12-16T07:44:49.938+0000 7fa8da62f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T07:44:51.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:50 smithi007 bash[30436]: audit 2023-12-16T07:44:49.573819+0000 mgr.y (mgr.15429) 9 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:51.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:50 smithi007 bash[30436]: audit 2023-12-16T07:44:49.788698+0000 mon.a (mon.0) 1046 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:51.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:50 smithi007 bash[30436]: audit 2023-12-16T07:44:49.796416+0000 mon.a (mon.0) 1047 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:51.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:50 smithi007 bash[30436]: audit 2023-12-16T07:44:50.595740+0000 mon.a (mon.0) 1048 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:51.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:50 smithi007 bash[30436]: audit 2023-12-16T07:44:50.603299+0000 mon.a (mon.0) 1049 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:51.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:50 smithi007 bash[25009]: audit 2023-12-16T07:44:49.573819+0000 mgr.y (mgr.15429) 9 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:51.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:50 smithi007 bash[25009]: audit 2023-12-16T07:44:49.788698+0000 mon.a (mon.0) 1046 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:51.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:50 smithi007 bash[25009]: audit 2023-12-16T07:44:49.796416+0000 mon.a (mon.0) 1047 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:51.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:50 smithi007 bash[25009]: audit 2023-12-16T07:44:50.595740+0000 mon.a (mon.0) 1048 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:51.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:50 smithi007 bash[25009]: audit 2023-12-16T07:44:50.603299+0000 mon.a (mon.0) 1049 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:51.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:50 smithi169 bash[25083]: audit 2023-12-16T07:44:49.573819+0000 mgr.y (mgr.15429) 9 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:44:51.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:50 smithi169 bash[25083]: audit 2023-12-16T07:44:49.788698+0000 mon.a (mon.0) 1046 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:51.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:50 smithi169 bash[25083]: audit 2023-12-16T07:44:49.796416+0000 mon.a (mon.0) 1047 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:51.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:50 smithi169 bash[25083]: audit 2023-12-16T07:44:50.595740+0000 mon.a (mon.0) 1048 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:51.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:50 smithi169 bash[25083]: audit 2023-12-16T07:44:50.603299+0000 mon.a (mon.0) 1049 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:51.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:51 smithi169 bash[45188]: debug 2023-12-16T07:44:51.186+0000 7fa8da62f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T07:44:51.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:51 smithi169 bash[45188]: [16/Dec/2023:07:44:51] ENGINE Bus STARTING 2023-12-16T07:44:51.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:51 smithi169 bash[45188]: CherryPy Checker: 2023-12-16T07:44:51.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:51 smithi169 bash[45188]: The Application mounted at '' has an empty config. 2023-12-16T07:44:51.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:51 smithi169 bash[45188]: [16/Dec/2023:07:44:51] ENGINE Serving on http://:::9283 2023-12-16T07:44:51.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:51 smithi169 bash[45188]: [16/Dec/2023:07:44:51] ENGINE Bus STARTED 2023-12-16T07:44:52.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[25009]: cluster 2023-12-16T07:44:50.693857+0000 mgr.y (mgr.15429) 10 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2023-12-16T07:44:52.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[25009]: audit 2023-12-16T07:44:50.797991+0000 mon.a (mon.0) 1050 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:52.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[25009]: audit 2023-12-16T07:44:50.805040+0000 mon.a (mon.0) 1051 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:52.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[25009]: cluster 2023-12-16T07:44:51.212659+0000 mon.a (mon.0) 1052 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:44:52.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[25009]: audit 2023-12-16T07:44:51.216987+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.? 172.21.15.169:0/3274532874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:44:52.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[25009]: audit 2023-12-16T07:44:51.220395+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.? 172.21.15.169:0/3274532874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:44:52.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[25009]: audit 2023-12-16T07:44:51.226491+0000 mon.b (mon.2) 88 : audit [DBG] from='mgr.? 172.21.15.169:0/3274532874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:44:52.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[25009]: audit 2023-12-16T07:44:51.227043+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.? 172.21.15.169:0/3274532874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:44:52.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[30436]: cluster 2023-12-16T07:44:50.693857+0000 mgr.y (mgr.15429) 10 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2023-12-16T07:44:52.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[30436]: audit 2023-12-16T07:44:50.797991+0000 mon.a (mon.0) 1050 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:52.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[30436]: audit 2023-12-16T07:44:50.805040+0000 mon.a (mon.0) 1051 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:52.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[30436]: cluster 2023-12-16T07:44:51.212659+0000 mon.a (mon.0) 1052 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:44:52.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[30436]: audit 2023-12-16T07:44:51.216987+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.? 172.21.15.169:0/3274532874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:44:52.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[30436]: audit 2023-12-16T07:44:51.220395+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.? 172.21.15.169:0/3274532874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:44:52.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[30436]: audit 2023-12-16T07:44:51.226491+0000 mon.b (mon.2) 88 : audit [DBG] from='mgr.? 172.21.15.169:0/3274532874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:44:52.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:51 smithi007 bash[30436]: audit 2023-12-16T07:44:51.227043+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.? 172.21.15.169:0/3274532874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:44:52.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:51 smithi169 bash[25083]: cluster 2023-12-16T07:44:50.693857+0000 mgr.y (mgr.15429) 10 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2023-12-16T07:44:52.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:51 smithi169 bash[25083]: audit 2023-12-16T07:44:50.797991+0000 mon.a (mon.0) 1050 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:52.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:51 smithi169 bash[25083]: audit 2023-12-16T07:44:50.805040+0000 mon.a (mon.0) 1051 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:44:52.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:51 smithi169 bash[25083]: cluster 2023-12-16T07:44:51.212659+0000 mon.a (mon.0) 1052 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:44:52.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:51 smithi169 bash[25083]: audit 2023-12-16T07:44:51.216987+0000 mon.b (mon.2) 86 : audit [DBG] from='mgr.? 172.21.15.169:0/3274532874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:44:52.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:51 smithi169 bash[25083]: audit 2023-12-16T07:44:51.220395+0000 mon.b (mon.2) 87 : audit [DBG] from='mgr.? 172.21.15.169:0/3274532874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:44:52.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:51 smithi169 bash[25083]: audit 2023-12-16T07:44:51.226491+0000 mon.b (mon.2) 88 : audit [DBG] from='mgr.? 172.21.15.169:0/3274532874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:44:52.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:51 smithi169 bash[25083]: audit 2023-12-16T07:44:51.227043+0000 mon.b (mon.2) 89 : audit [DBG] from='mgr.? 172.21.15.169:0/3274532874' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:44:53.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:52 smithi007 bash[30436]: cluster 2023-12-16T07:44:51.818128+0000 mon.a (mon.0) 1053 : cluster [DBG] mgrmap e32: y(active, since 10s), standbys: x 2023-12-16T07:44:53.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:52 smithi007 bash[30436]: audit 2023-12-16T07:44:51.819177+0000 mon.a (mon.0) 1054 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:44:53.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:52 smithi007 bash[30436]: cluster 2023-12-16T07:44:52.694593+0000 mgr.y (mgr.15429) 11 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2023-12-16T07:44:53.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:52 smithi007 bash[25009]: cluster 2023-12-16T07:44:51.818128+0000 mon.a (mon.0) 1053 : cluster [DBG] mgrmap e32: y(active, since 10s), standbys: x 2023-12-16T07:44:53.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:52 smithi007 bash[25009]: audit 2023-12-16T07:44:51.819177+0000 mon.a (mon.0) 1054 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:44:53.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:52 smithi007 bash[25009]: cluster 2023-12-16T07:44:52.694593+0000 mgr.y (mgr.15429) 11 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2023-12-16T07:44:53.085 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:44:53.127 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:52 smithi169 bash[25083]: cluster 2023-12-16T07:44:51.818128+0000 mon.a (mon.0) 1053 : cluster [DBG] mgrmap e32: y(active, since 10s), standbys: x 2023-12-16T07:44:53.127 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:52 smithi169 bash[25083]: audit 2023-12-16T07:44:51.819177+0000 mon.a (mon.0) 1054 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:44:53.127 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:52 smithi169 bash[25083]: cluster 2023-12-16T07:44:52.694593+0000 mgr.y (mgr.15429) 11 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2023-12-16T07:44:53.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:44:53 smithi169 bash[47208]: ts=2023-12-16T07:44:53.126Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.169:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.169:8765: connect: connection refused" 2023-12-16T07:44:53.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:44:53 smithi169 bash[47208]: ts=2023-12-16T07:44:53.130Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.169:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.169:8765: connect: connection refused" 2023-12-16T07:44:53.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:44:53 smithi169 bash[47208]: ts=2023-12-16T07:44:53.130Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.169:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.169:8765: connect: connection refused" 2023-12-16T07:44:53.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:44:53 smithi169 bash[47208]: ts=2023-12-16T07:44:53.130Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.169:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.169:8765: connect: connection refused" 2023-12-16T07:44:53.869 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:44:53.869 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (81s) 4s ago 8m 14.6M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:44:53.869 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (36s) 4s ago 8m 42.6M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:44:53.869 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (76s) 4s ago 6m 46.8M - 3.5 e1d6a67b021e 6f450ed9dcaa 2023-12-16T07:44:53.869 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283 running (2m) 4s ago 14m 175M - 19.0.0-191-g6d933094 a29c258b5d6f e5055e9fae4f 2023-12-16T07:44:53.869 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (30s) 4s ago 16m 467M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:44:53.869 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (16m) 4s ago 16m 66.0M 2048M 17.2.0 e1d6a67b021e c0468ecbd44e 2023-12-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (14m) 4s ago 14m 56.2M 2048M 17.2.0 e1d6a67b021e b47a960c6102 2023-12-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (14m) 4s ago 14m 51.1M 2048M 17.2.0 e1d6a67b021e c659b986f201 2023-12-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (91s) 4s ago 8m 10.8M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (60s) 4s ago 8m 13.8M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (13m) 4s ago 14m 57.5M 2782M 17.2.0 e1d6a67b021e dbc4d9538238 2023-12-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (13m) 4s ago 13m 58.3M 2782M 17.2.0 e1d6a67b021e 14637e0f389c 2023-12-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (12m) 4s ago 12m 56.5M 2782M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (12m) 4s ago 12m 56.3M 2782M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (11m) 4s ago 11m 57.6M 3037M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (10m) 4s ago 10m 54.2M 3037M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (10m) 4s ago 10m 54.5M 3037M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (9m) 4s ago 9m 55.4M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (62s) 4s ago 8m 50.6M - 2.33.4 514e6a882f6e 399a7aff3d32 2023-12-16T07:44:53.870 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (6m) 4s ago 6m 86.2M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:44:53.871 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (6m) 4s ago 6m 86.0M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:44:53.871 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (6m) 4s ago 6m 85.8M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:44:53.871 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (6m) 4s ago 6m 86.1M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:44:54.333 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:44:54.333 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:44:54.333 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2023-12-16T07:44:54.333 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:44:54.333 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:44:54.333 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:44:54.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:44:54.334 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:44:54.334 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2023-12-16T07:44:54.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:44:54.334 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2023-12-16T07:44:54.334 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:44:54.334 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:44:54.334 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:44:54.334 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:44:54.334 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2023-12-16T07:44:54.334 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:44:54.334 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:44:54.334 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:44:54.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:54 smithi007 bash[30436]: audit 2023-12-16T07:44:53.066942+0000 mgr.y (mgr.15429) 12 : audit [DBG] from='client.15465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:54.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:54 smithi007 bash[30436]: audit 2023-12-16T07:44:53.457046+0000 mgr.y (mgr.15429) 13 : audit [DBG] from='client.25378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:54.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:54 smithi007 bash[25009]: audit 2023-12-16T07:44:53.066942+0000 mgr.y (mgr.15429) 12 : audit [DBG] from='client.15465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:54.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:54 smithi007 bash[25009]: audit 2023-12-16T07:44:53.457046+0000 mgr.y (mgr.15429) 13 : audit [DBG] from='client.25378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:54.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:54 smithi169 bash[25083]: audit 2023-12-16T07:44:53.066942+0000 mgr.y (mgr.15429) 12 : audit [DBG] from='client.15465 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:54.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:54 smithi169 bash[25083]: audit 2023-12-16T07:44:53.457046+0000 mgr.y (mgr.15429) 13 : audit [DBG] from='client.25378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:44:54.716 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2023-12-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2023-12-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "2/23 daemons upgraded", 2023-12-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "", 2023-12-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:44:54.717 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:44:55.177 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:44:55.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:55 smithi007 bash[25009]: audit 2023-12-16T07:44:53.841847+0000 mgr.y (mgr.15429) 14 : audit [DBG] from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:55.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:55 smithi007 bash[25009]: audit 2023-12-16T07:44:54.330561+0000 mon.c (mon.1) 45 : audit [DBG] from='client.? 172.21.15.7:0/1120707919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:44:55.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:55 smithi007 bash[25009]: cluster 2023-12-16T07:44:54.696226+0000 mgr.y (mgr.15429) 15 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2023-12-16T07:44:55.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:55 smithi007 bash[25009]: audit 2023-12-16T07:44:54.714132+0000 mgr.y (mgr.15429) 16 : audit [DBG] from='client.15489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:55.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:55 smithi007 bash[25009]: audit 2023-12-16T07:44:55.175223+0000 mon.c (mon.1) 46 : audit [DBG] from='client.? 172.21.15.7:0/1201961836' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:44:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:55 smithi007 bash[30436]: audit 2023-12-16T07:44:53.841847+0000 mgr.y (mgr.15429) 14 : audit [DBG] from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:55 smithi007 bash[30436]: audit 2023-12-16T07:44:54.330561+0000 mon.c (mon.1) 45 : audit [DBG] from='client.? 172.21.15.7:0/1120707919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:44:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:55 smithi007 bash[30436]: cluster 2023-12-16T07:44:54.696226+0000 mgr.y (mgr.15429) 15 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2023-12-16T07:44:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:55 smithi007 bash[30436]: audit 2023-12-16T07:44:54.714132+0000 mgr.y (mgr.15429) 16 : audit [DBG] from='client.15489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:55 smithi007 bash[30436]: audit 2023-12-16T07:44:55.175223+0000 mon.c (mon.1) 46 : audit [DBG] from='client.? 172.21.15.7:0/1201961836' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:44:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:55 smithi169 bash[25083]: audit 2023-12-16T07:44:53.841847+0000 mgr.y (mgr.15429) 14 : audit [DBG] from='client.15477 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:55 smithi169 bash[25083]: audit 2023-12-16T07:44:54.330561+0000 mon.c (mon.1) 45 : audit [DBG] from='client.? 172.21.15.7:0/1120707919' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:44:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:55 smithi169 bash[25083]: cluster 2023-12-16T07:44:54.696226+0000 mgr.y (mgr.15429) 15 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2023-12-16T07:44:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:55 smithi169 bash[25083]: audit 2023-12-16T07:44:54.714132+0000 mgr.y (mgr.15429) 16 : audit [DBG] from='client.15489 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:44:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:55 smithi169 bash[25083]: audit 2023-12-16T07:44:55.175223+0000 mon.c (mon.1) 46 : audit [DBG] from='client.? 172.21.15.7:0/1201961836' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:44:58.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:57 smithi007 bash[25009]: cluster 2023-12-16T07:44:56.696985+0000 mgr.y (mgr.15429) 17 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:44:58.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:57 smithi007 bash[30436]: cluster 2023-12-16T07:44:56.696985+0000 mgr.y (mgr.15429) 17 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:44:58.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:57 smithi169 bash[25083]: cluster 2023-12-16T07:44:56.696985+0000 mgr.y (mgr.15429) 17 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:44:59.020 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:58 smithi007 bash[25009]: audit 2023-12-16T07:44:57.852540+0000 mon.a (mon.0) 1055 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:59.020 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:58 smithi007 bash[30436]: audit 2023-12-16T07:44:57.852540+0000 mon.a (mon.0) 1055 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:59.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:58 smithi169 bash[25083]: audit 2023-12-16T07:44:57.852540+0000 mon.a (mon.0) 1055 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:44:59.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:44:58 smithi169 bash[45188]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:44:58] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:45:00.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:44:59 smithi007 bash[30436]: cluster 2023-12-16T07:44:58.697937+0000 mgr.y (mgr.15429) 18 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:45:00.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:44:59 smithi007 bash[25009]: cluster 2023-12-16T07:44:58.697937+0000 mgr.y (mgr.15429) 18 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:45:00.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:44:59 smithi169 bash[25083]: cluster 2023-12-16T07:44:58.697937+0000 mgr.y (mgr.15429) 18 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:45:00.779 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:00 smithi007 bash[30436]: audit 2023-12-16T07:44:59.583215+0000 mgr.y (mgr.15429) 19 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:01.031 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:00 smithi169 bash[25083]: audit 2023-12-16T07:44:59.583215+0000 mgr.y (mgr.15429) 19 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:01.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:00 smithi007 bash[25009]: audit 2023-12-16T07:44:59.583215+0000 mgr.y (mgr.15429) 19 : audit [DBG] from='client.25204 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:02.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:01 smithi007 bash[25009]: cluster 2023-12-16T07:45:00.699751+0000 mgr.y (mgr.15429) 20 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:45:02.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:01 smithi007 bash[25009]: audit 2023-12-16T07:45:01.157731+0000 mon.a (mon.0) 1056 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:02.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:01 smithi007 bash[25009]: audit 2023-12-16T07:45:01.170049+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:02.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:01 smithi007 bash[25009]: audit 2023-12-16T07:45:01.173004+0000 mon.a (mon.0) 1058 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:02.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:01 smithi007 bash[25009]: audit 2023-12-16T07:45:01.175290+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:45:02.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:01 smithi007 bash[30436]: cluster 2023-12-16T07:45:00.699751+0000 mgr.y (mgr.15429) 20 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:45:02.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:01 smithi007 bash[30436]: audit 2023-12-16T07:45:01.157731+0000 mon.a (mon.0) 1056 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:02.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:01 smithi007 bash[30436]: audit 2023-12-16T07:45:01.170049+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:02.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:01 smithi007 bash[30436]: audit 2023-12-16T07:45:01.173004+0000 mon.a (mon.0) 1058 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:02.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:01 smithi007 bash[30436]: audit 2023-12-16T07:45:01.175290+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:45:02.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:01 smithi169 bash[25083]: cluster 2023-12-16T07:45:00.699751+0000 mgr.y (mgr.15429) 20 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:45:02.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:01 smithi169 bash[25083]: audit 2023-12-16T07:45:01.157731+0000 mon.a (mon.0) 1056 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:02.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:01 smithi169 bash[25083]: audit 2023-12-16T07:45:01.170049+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:02.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:01 smithi169 bash[25083]: audit 2023-12-16T07:45:01.173004+0000 mon.a (mon.0) 1058 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:02.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:01 smithi169 bash[25083]: audit 2023-12-16T07:45:01.175290+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:45:03.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: cephadm 2023-12-16T07:45:01.177236+0000 mgr.y (mgr.15429) 21 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:45:03.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: cephadm 2023-12-16T07:45:01.177537+0000 mgr.y (mgr.15429) 22 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:45:03.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: cephadm 2023-12-16T07:45:01.342174+0000 mgr.y (mgr.15429) 23 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:45:03.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: cephadm 2023-12-16T07:45:01.343008+0000 mgr.y (mgr.15429) 24 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:45:03.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: cephadm 2023-12-16T07:45:01.507144+0000 mgr.y (mgr.15429) 25 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:45:03.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: cephadm 2023-12-16T07:45:01.507939+0000 mgr.y (mgr.15429) 26 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:45:03.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: cephadm 2023-12-16T07:45:01.671884+0000 mgr.y (mgr.15429) 27 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:45:03.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: cephadm 2023-12-16T07:45:01.673392+0000 mgr.y (mgr.15429) 28 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:45:03.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: audit 2023-12-16T07:45:01.846144+0000 mon.a (mon.0) 1060 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: audit 2023-12-16T07:45:01.852575+0000 mon.a (mon.0) 1061 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: audit 2023-12-16T07:45:01.857904+0000 mon.a (mon.0) 1062 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: audit 2023-12-16T07:45:01.862482+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: audit 2023-12-16T07:45:01.869892+0000 mon.a (mon.0) 1064 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: audit 2023-12-16T07:45:01.944657+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-12-16T07:45:03.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[30436]: audit 2023-12-16T07:45:01.967871+0000 mon.a (mon.0) 1066 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:03.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: cephadm 2023-12-16T07:45:01.177236+0000 mgr.y (mgr.15429) 21 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:45:03.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: cephadm 2023-12-16T07:45:01.177537+0000 mgr.y (mgr.15429) 22 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:45:03.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: cephadm 2023-12-16T07:45:01.342174+0000 mgr.y (mgr.15429) 23 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:45:03.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: cephadm 2023-12-16T07:45:01.343008+0000 mgr.y (mgr.15429) 24 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:45:03.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: cephadm 2023-12-16T07:45:01.507144+0000 mgr.y (mgr.15429) 25 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:45:03.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: cephadm 2023-12-16T07:45:01.507939+0000 mgr.y (mgr.15429) 26 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:45:03.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: cephadm 2023-12-16T07:45:01.671884+0000 mgr.y (mgr.15429) 27 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:45:03.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: cephadm 2023-12-16T07:45:01.673392+0000 mgr.y (mgr.15429) 28 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:45:03.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: audit 2023-12-16T07:45:01.846144+0000 mon.a (mon.0) 1060 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: audit 2023-12-16T07:45:01.852575+0000 mon.a (mon.0) 1061 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: audit 2023-12-16T07:45:01.857904+0000 mon.a (mon.0) 1062 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: audit 2023-12-16T07:45:01.862482+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: audit 2023-12-16T07:45:01.869892+0000 mon.a (mon.0) 1064 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: audit 2023-12-16T07:45:01.944657+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-12-16T07:45:03.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:02 smithi007 bash[25009]: audit 2023-12-16T07:45:01.967871+0000 mon.a (mon.0) 1066 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:03.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: cephadm 2023-12-16T07:45:01.177236+0000 mgr.y (mgr.15429) 21 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:45:03.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: cephadm 2023-12-16T07:45:01.177537+0000 mgr.y (mgr.15429) 22 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:45:03.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: cephadm 2023-12-16T07:45:01.342174+0000 mgr.y (mgr.15429) 23 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:45:03.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: cephadm 2023-12-16T07:45:01.343008+0000 mgr.y (mgr.15429) 24 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:45:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: cephadm 2023-12-16T07:45:01.507144+0000 mgr.y (mgr.15429) 25 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:45:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: cephadm 2023-12-16T07:45:01.507939+0000 mgr.y (mgr.15429) 26 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:45:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: cephadm 2023-12-16T07:45:01.671884+0000 mgr.y (mgr.15429) 27 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:45:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: cephadm 2023-12-16T07:45:01.673392+0000 mgr.y (mgr.15429) 28 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:45:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: audit 2023-12-16T07:45:01.846144+0000 mon.a (mon.0) 1060 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: audit 2023-12-16T07:45:01.852575+0000 mon.a (mon.0) 1061 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: audit 2023-12-16T07:45:01.857904+0000 mon.a (mon.0) 1062 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: audit 2023-12-16T07:45:01.862482+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: audit 2023-12-16T07:45:01.869892+0000 mon.a (mon.0) 1064 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: audit 2023-12-16T07:45:01.944657+0000 mon.a (mon.0) 1065 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.crrdaq", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-12-16T07:45:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:02 smithi169 bash[25083]: audit 2023-12-16T07:45:01.967871+0000 mon.a (mon.0) 1066 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:04.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:03 smithi007 bash[25009]: cephadm 2023-12-16T07:45:01.943683+0000 mgr.y (mgr.15429) 29 : cephadm [INF] Reconfiguring iscsi.foo.smithi007.crrdaq (dependencies changed)... 2023-12-16T07:45:04.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:03 smithi007 bash[25009]: cephadm 2023-12-16T07:45:01.969922+0000 mgr.y (mgr.15429) 30 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi007.crrdaq on smithi007 2023-12-16T07:45:04.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:03 smithi007 bash[25009]: cluster 2023-12-16T07:45:02.700542+0000 mgr.y (mgr.15429) 31 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:04.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:03 smithi007 bash[30436]: cephadm 2023-12-16T07:45:01.943683+0000 mgr.y (mgr.15429) 29 : cephadm [INF] Reconfiguring iscsi.foo.smithi007.crrdaq (dependencies changed)... 2023-12-16T07:45:04.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:03 smithi007 bash[30436]: cephadm 2023-12-16T07:45:01.969922+0000 mgr.y (mgr.15429) 30 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi007.crrdaq on smithi007 2023-12-16T07:45:04.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:03 smithi007 bash[30436]: cluster 2023-12-16T07:45:02.700542+0000 mgr.y (mgr.15429) 31 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:04.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:03 smithi169 bash[25083]: cephadm 2023-12-16T07:45:01.943683+0000 mgr.y (mgr.15429) 29 : cephadm [INF] Reconfiguring iscsi.foo.smithi007.crrdaq (dependencies changed)... 2023-12-16T07:45:04.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:03 smithi169 bash[25083]: cephadm 2023-12-16T07:45:01.969922+0000 mgr.y (mgr.15429) 30 : cephadm [INF] Reconfiguring daemon iscsi.foo.smithi007.crrdaq on smithi007 2023-12-16T07:45:04.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:03 smithi169 bash[25083]: cluster 2023-12-16T07:45:02.700542+0000 mgr.y (mgr.15429) 31 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:06.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:05 smithi007 bash[30436]: cluster 2023-12-16T07:45:04.702215+0000 mgr.y (mgr.15429) 32 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:45:06.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:05 smithi007 bash[25009]: cluster 2023-12-16T07:45:04.702215+0000 mgr.y (mgr.15429) 32 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:45:06.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:05 smithi169 bash[25083]: cluster 2023-12-16T07:45:04.702215+0000 mgr.y (mgr.15429) 32 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:45:07.226 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:06 smithi169 bash[25083]: audit 2023-12-16T07:45:05.954235+0000 mon.a (mon.0) 1067 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:07.226 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:06 smithi169 bash[25083]: audit 2023-12-16T07:45:05.962152+0000 mon.a (mon.0) 1068 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:07.226 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:06 smithi169 bash[25083]: cephadm 2023-12-16T07:45:06.005708+0000 mgr.y (mgr.15429) 33 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-12-16T07:45:07.227 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:06 smithi169 bash[25083]: cephadm 2023-12-16T07:45:06.273949+0000 mgr.y (mgr.15429) 34 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi169 2023-12-16T07:45:07.227 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:06 smithi169 bash[25083]: cluster 2023-12-16T07:45:06.703197+0000 mgr.y (mgr.15429) 35 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:07.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:06 smithi007 bash[30436]: audit 2023-12-16T07:45:05.954235+0000 mon.a (mon.0) 1067 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:07.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:06 smithi007 bash[30436]: audit 2023-12-16T07:45:05.962152+0000 mon.a (mon.0) 1068 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:07.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:06 smithi007 bash[30436]: cephadm 2023-12-16T07:45:06.005708+0000 mgr.y (mgr.15429) 33 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-12-16T07:45:07.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:06 smithi007 bash[30436]: cephadm 2023-12-16T07:45:06.273949+0000 mgr.y (mgr.15429) 34 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi169 2023-12-16T07:45:07.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:06 smithi007 bash[30436]: cluster 2023-12-16T07:45:06.703197+0000 mgr.y (mgr.15429) 35 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:07.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:06 smithi007 bash[25009]: audit 2023-12-16T07:45:05.954235+0000 mon.a (mon.0) 1067 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:07.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:06 smithi007 bash[25009]: audit 2023-12-16T07:45:05.962152+0000 mon.a (mon.0) 1068 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:07.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:06 smithi007 bash[25009]: cephadm 2023-12-16T07:45:06.005708+0000 mgr.y (mgr.15429) 33 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2023-12-16T07:45:07.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:06 smithi007 bash[25009]: cephadm 2023-12-16T07:45:06.273949+0000 mgr.y (mgr.15429) 34 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi169 2023-12-16T07:45:07.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:06 smithi007 bash[25009]: cluster 2023-12-16T07:45:06.703197+0000 mgr.y (mgr.15429) 35 : cluster [DBG] pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:08.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:08 smithi007 bash[25009]: audit 2023-12-16T07:45:08.522052+0000 mon.a (mon.0) 1069 : audit [DBG] from='client.? 172.21.15.7:0/2262690160' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-12-16T07:45:08.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:08 smithi007 bash[30436]: audit 2023-12-16T07:45:08.522052+0000 mon.a (mon.0) 1069 : audit [DBG] from='client.? 172.21.15.7:0/2262690160' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-12-16T07:45:08.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:08 smithi169 bash[25083]: audit 2023-12-16T07:45:08.522052+0000 mon.a (mon.0) 1069 : audit [DBG] from='client.? 172.21.15.7:0/2262690160' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-12-16T07:45:08.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:08 smithi169 bash[45188]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:45:08] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2023-12-16T07:45:09.891 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:09 smithi169 bash[25083]: cluster 2023-12-16T07:45:08.704221+0000 mgr.y (mgr.15429) 36 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:10.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:09 smithi007 bash[25009]: cluster 2023-12-16T07:45:08.704221+0000 mgr.y (mgr.15429) 36 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:10.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:09 smithi007 bash[30436]: cluster 2023-12-16T07:45:08.704221+0000 mgr.y (mgr.15429) 36 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:10.851 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:10 smithi169 systemd[1]: Stopping Ceph prometheus.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:45:10.851 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:10 smithi169 bash[49014]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-prometheus.a 2023-12-16T07:45:10.851 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:10 smithi169 bash[47208]: ts=2023-12-16T07:45:10.691Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2023-12-16T07:45:10.851 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:10 smithi169 bash[47208]: ts=2023-12-16T07:45:10.691Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2023-12-16T07:45:10.851 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:10 smithi169 bash[47208]: ts=2023-12-16T07:45:10.691Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2023-12-16T07:45:10.851 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:10 smithi169 bash[47208]: ts=2023-12-16T07:45:10.691Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2023-12-16T07:45:10.851 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:10 smithi169 bash[47208]: ts=2023-12-16T07:45:10.691Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2023-12-16T07:45:10.851 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:10 smithi169 bash[47208]: ts=2023-12-16T07:45:10.691Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2023-12-16T07:45:10.851 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:10 smithi169 bash[47208]: ts=2023-12-16T07:45:10.692Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2023-12-16T07:45:10.852 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:10 smithi169 bash[47208]: ts=2023-12-16T07:45:10.692Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2023-12-16T07:45:10.852 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:10 smithi169 bash[47208]: ts=2023-12-16T07:45:10.692Z caller=main.go:828 level=info msg="Scrape manager stopped" 2023-12-16T07:45:10.852 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:10 smithi169 bash[47208]: ts=2023-12-16T07:45:10.692Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2023-12-16T07:45:10.852 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:10 smithi169 bash[47208]: ts=2023-12-16T07:45:10.692Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2023-12-16T07:45:10.852 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:10 smithi169 bash[47208]: ts=2023-12-16T07:45:10.692Z caller=main.go:1066 level=info msg="See you next time!" 2023-12-16T07:45:11.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:11 smithi169 bash[49022]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-prometheus-a 2023-12-16T07:45:11.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:11 smithi169 bash[49058]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-prometheus.a 2023-12-16T07:45:11.434 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:11 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@prometheus.a.service: Succeeded. 2023-12-16T07:45:11.434 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:11 smithi169 systemd[1]: Stopped Ceph prometheus.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:45:11.434 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:11 smithi169 systemd[1]: Started Ceph prometheus.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:45:11.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[25009]: cluster 2023-12-16T07:45:10.705845+0000 mgr.y (mgr.15429) 37 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-12-16T07:45:11.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[25009]: audit 2023-12-16T07:45:11.456960+0000 mon.a (mon.0) 1070 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:11.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[25009]: audit 2023-12-16T07:45:11.466954+0000 mon.a (mon.0) 1071 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:11.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[25009]: audit 2023-12-16T07:45:11.510003+0000 mon.a (mon.0) 1072 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:45:11.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[25009]: audit 2023-12-16T07:45:11.531548+0000 mon.a (mon.0) 1073 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:11.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[25009]: audit 2023-12-16T07:45:11.534623+0000 mon.a (mon.0) 1074 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:45:11.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[25009]: audit 2023-12-16T07:45:11.538254+0000 mon.a (mon.0) 1075 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:45:11.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[25009]: audit 2023-12-16T07:45:11.545748+0000 mon.a (mon.0) 1076 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:11.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[25009]: audit 2023-12-16T07:45:11.553762+0000 mon.a (mon.0) 1077 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:45:11.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[25009]: audit 2023-12-16T07:45:11.564248+0000 mon.a (mon.0) 1078 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:45:11.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[30436]: cluster 2023-12-16T07:45:10.705845+0000 mgr.y (mgr.15429) 37 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-12-16T07:45:11.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[30436]: audit 2023-12-16T07:45:11.456960+0000 mon.a (mon.0) 1070 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:11.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[30436]: audit 2023-12-16T07:45:11.466954+0000 mon.a (mon.0) 1071 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:11.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[30436]: audit 2023-12-16T07:45:11.510003+0000 mon.a (mon.0) 1072 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:45:11.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[30436]: audit 2023-12-16T07:45:11.531548+0000 mon.a (mon.0) 1073 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:11.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[30436]: audit 2023-12-16T07:45:11.534623+0000 mon.a (mon.0) 1074 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:45:11.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[30436]: audit 2023-12-16T07:45:11.538254+0000 mon.a (mon.0) 1075 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:45:11.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[30436]: audit 2023-12-16T07:45:11.545748+0000 mon.a (mon.0) 1076 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:11.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[30436]: audit 2023-12-16T07:45:11.553762+0000 mon.a (mon.0) 1077 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:45:11.824 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:11 smithi007 bash[30436]: audit 2023-12-16T07:45:11.564248+0000 mon.a (mon.0) 1078 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:45:12.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:11 smithi169 bash[25083]: cluster 2023-12-16T07:45:10.705845+0000 mgr.y (mgr.15429) 37 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-12-16T07:45:12.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:11 smithi169 bash[25083]: audit 2023-12-16T07:45:11.456960+0000 mon.a (mon.0) 1070 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:12.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:11 smithi169 bash[25083]: audit 2023-12-16T07:45:11.466954+0000 mon.a (mon.0) 1071 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:12.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:11 smithi169 bash[25083]: audit 2023-12-16T07:45:11.510003+0000 mon.a (mon.0) 1072 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:45:12.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:11 smithi169 bash[25083]: audit 2023-12-16T07:45:11.531548+0000 mon.a (mon.0) 1073 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:12.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:11 smithi169 bash[25083]: audit 2023-12-16T07:45:11.534623+0000 mon.a (mon.0) 1074 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:45:12.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:11 smithi169 bash[25083]: audit 2023-12-16T07:45:11.538254+0000 mon.a (mon.0) 1075 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:45:12.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:11 smithi169 bash[25083]: audit 2023-12-16T07:45:11.545748+0000 mon.a (mon.0) 1076 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:12.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:11 smithi169 bash[25083]: audit 2023-12-16T07:45:11.553762+0000 mon.a (mon.0) 1077 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:45:12.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:11 smithi169 bash[25083]: audit 2023-12-16T07:45:11.564248+0000 mon.a (mon.0) 1078 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:45:13.028 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:12 smithi169 bash[25083]: audit 2023-12-16T07:45:11.511629+0000 mgr.y (mgr.15429) 38 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:45:13.029 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:12 smithi169 bash[25083]: cephadm 2023-12-16T07:45:11.533997+0000 mgr.y (mgr.15429) 39 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2023-12-16T07:45:13.029 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:12 smithi169 bash[25083]: audit 2023-12-16T07:45:11.535451+0000 mgr.y (mgr.15429) 40 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:45:13.029 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:12 smithi169 bash[25083]: audit 2023-12-16T07:45:11.539303+0000 mgr.y (mgr.15429) 41 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:45:13.029 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:12 smithi169 bash[25083]: audit 2023-12-16T07:45:11.554696+0000 mgr.y (mgr.15429) 42 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:45:13.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:12 smithi007 bash[30436]: audit 2023-12-16T07:45:11.511629+0000 mgr.y (mgr.15429) 38 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:45:13.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:12 smithi007 bash[30436]: cephadm 2023-12-16T07:45:11.533997+0000 mgr.y (mgr.15429) 39 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2023-12-16T07:45:13.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:12 smithi007 bash[30436]: audit 2023-12-16T07:45:11.535451+0000 mgr.y (mgr.15429) 40 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:45:13.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:12 smithi007 bash[30436]: audit 2023-12-16T07:45:11.539303+0000 mgr.y (mgr.15429) 41 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:45:13.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:12 smithi007 bash[30436]: audit 2023-12-16T07:45:11.554696+0000 mgr.y (mgr.15429) 42 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:45:13.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:12 smithi007 bash[25009]: audit 2023-12-16T07:45:11.511629+0000 mgr.y (mgr.15429) 38 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2023-12-16T07:45:13.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:12 smithi007 bash[25009]: cephadm 2023-12-16T07:45:11.533997+0000 mgr.y (mgr.15429) 39 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2023-12-16T07:45:13.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:12 smithi007 bash[25009]: audit 2023-12-16T07:45:11.535451+0000 mgr.y (mgr.15429) 40 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2023-12-16T07:45:13.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:12 smithi007 bash[25009]: audit 2023-12-16T07:45:11.539303+0000 mgr.y (mgr.15429) 41 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2023-12-16T07:45:13.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:12 smithi007 bash[25009]: audit 2023-12-16T07:45:11.554696+0000 mgr.y (mgr.15429) 42 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2023-12-16T07:45:13.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.102Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2023-12-16T07:45:13.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.102Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2023-12-16T07:45:13.433 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.102Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2023-12-16T07:45:13.434 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.102Z caller=main.go:518 level=info host_details="(Linux 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 smithi169 (none))" 2023-12-16T07:45:13.434 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.102Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2023-12-16T07:45:13.434 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.102Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2023-12-16T07:45:13.434 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.104Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2023-12-16T07:45:13.434 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.105Z caller=main.go:923 level=info msg="Starting TSDB ..." 2023-12-16T07:45:13.434 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.107Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2023-12-16T07:45:13.434 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.108Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2023-12-16T07:45:13.434 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.108Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.673µs 2023-12-16T07:45:13.434 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.108Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2023-12-16T07:45:13.434 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.302Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2023-12-16T07:45:13.772 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.537Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2023-12-16T07:45:13.772 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.590Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2023-12-16T07:45:13.772 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.591Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2023-12-16T07:45:13.772 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.591Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=51.809µs wal_replay_duration=482.982738ms total_replay_duration=483.05267ms 2023-12-16T07:45:13.772 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.599Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2023-12-16T07:45:13.772 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.599Z caller=main.go:947 level=info msg="TSDB started" 2023-12-16T07:45:13.772 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.600Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2023-12-16T07:45:13.772 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.636Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=36.948365ms db_storage=1.567µs remote_storage=2.64µs web_handler=636ns query_engine=1.494µs scrape=707.059µs scrape_sd=243.975µs notify=28.95µs notify_sd=24.062µs rules=35.230695ms 2023-12-16T07:45:13.773 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[49087]: ts=2023-12-16T07:45:13.637Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2023-12-16T07:45:14.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:13 smithi007 bash[25009]: cluster 2023-12-16T07:45:12.706629+0000 mgr.y (mgr.15429) 43 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2023-12-16T07:45:14.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:13 smithi007 bash[25009]: audit 2023-12-16T07:45:12.852816+0000 mon.a (mon.0) 1079 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:45:14.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:13 smithi007 bash[30436]: cluster 2023-12-16T07:45:12.706629+0000 mgr.y (mgr.15429) 43 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2023-12-16T07:45:14.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:13 smithi007 bash[30436]: audit 2023-12-16T07:45:12.852816+0000 mon.a (mon.0) 1079 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:45:14.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[25083]: cluster 2023-12-16T07:45:12.706629+0000 mgr.y (mgr.15429) 43 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2023-12-16T07:45:14.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:13 smithi169 bash[25083]: audit 2023-12-16T07:45:12.852816+0000 mon.a (mon.0) 1079 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:45:14.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:14 smithi169 bash[25083]: audit 2023-12-16T07:45:14.040444+0000 mon.a (mon.0) 1080 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:45:14.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:14 smithi169 bash[25083]: audit 2023-12-16T07:45:14.042584+0000 mon.a (mon.0) 1081 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:45:14.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:14 smithi169 bash[25083]: audit 2023-12-16T07:45:14.044602+0000 mon.a (mon.0) 1082 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:15.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:14 smithi007 bash[25009]: audit 2023-12-16T07:45:14.040444+0000 mon.a (mon.0) 1080 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:45:15.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:14 smithi007 bash[25009]: audit 2023-12-16T07:45:14.042584+0000 mon.a (mon.0) 1081 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:45:15.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:14 smithi007 bash[25009]: audit 2023-12-16T07:45:14.044602+0000 mon.a (mon.0) 1082 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:15.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:14 smithi007 bash[30436]: audit 2023-12-16T07:45:14.040444+0000 mon.a (mon.0) 1080 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:45:15.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:14 smithi007 bash[30436]: audit 2023-12-16T07:45:14.042584+0000 mon.a (mon.0) 1081 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:45:15.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:14 smithi007 bash[30436]: audit 2023-12-16T07:45:14.044602+0000 mon.a (mon.0) 1082 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:16.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:15 smithi007 bash[25009]: cephadm 2023-12-16T07:45:14.039620+0000 mgr.y (mgr.15429) 44 : cephadm [INF] Upgrade: Updating mgr.x 2023-12-16T07:45:16.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:15 smithi007 bash[25009]: cephadm 2023-12-16T07:45:14.047015+0000 mgr.y (mgr.15429) 45 : cephadm [INF] Deploying daemon mgr.x on smithi169 2023-12-16T07:45:16.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:15 smithi007 bash[25009]: cluster 2023-12-16T07:45:14.708417+0000 mgr.y (mgr.15429) 46 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-12-16T07:45:16.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:15 smithi007 bash[30436]: cephadm 2023-12-16T07:45:14.039620+0000 mgr.y (mgr.15429) 44 : cephadm [INF] Upgrade: Updating mgr.x 2023-12-16T07:45:16.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:15 smithi007 bash[30436]: cephadm 2023-12-16T07:45:14.047015+0000 mgr.y (mgr.15429) 45 : cephadm [INF] Deploying daemon mgr.x on smithi169 2023-12-16T07:45:16.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:15 smithi007 bash[30436]: cluster 2023-12-16T07:45:14.708417+0000 mgr.y (mgr.15429) 46 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-12-16T07:45:16.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:15 smithi169 bash[25083]: cephadm 2023-12-16T07:45:14.039620+0000 mgr.y (mgr.15429) 44 : cephadm [INF] Upgrade: Updating mgr.x 2023-12-16T07:45:16.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:15 smithi169 bash[25083]: cephadm 2023-12-16T07:45:14.047015+0000 mgr.y (mgr.15429) 45 : cephadm [INF] Deploying daemon mgr.x on smithi169 2023-12-16T07:45:16.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:15 smithi169 bash[25083]: cluster 2023-12-16T07:45:14.708417+0000 mgr.y (mgr.15429) 46 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-12-16T07:45:16.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:16 smithi169 systemd[1]: Stopping Ceph mgr.x for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:45:17.553 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:17 smithi169 bash[49390]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mgr-x 2023-12-16T07:45:17.553 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:17 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2023-12-16T07:45:17.553 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:17 smithi169 bash[49472]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mgr-x 2023-12-16T07:45:17.553 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:17 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2023-12-16T07:45:17.553 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:17 smithi169 systemd[1]: Stopped Ceph mgr.x for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:45:17.843 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:17 smithi169 bash[25083]: cluster 2023-12-16T07:45:16.709377+0000 mgr.y (mgr.15429) 47 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2023-12-16T07:45:17.844 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:17 smithi169 systemd[1]: Started Ceph mgr.x for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:45:18.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:17 smithi007 bash[30436]: cluster 2023-12-16T07:45:16.709377+0000 mgr.y (mgr.15429) 47 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2023-12-16T07:45:18.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:17 smithi007 bash[25009]: cluster 2023-12-16T07:45:16.709377+0000 mgr.y (mgr.15429) 47 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2023-12-16T07:45:18.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:18 smithi169 bash[25083]: audit 2023-12-16T07:45:17.837446+0000 mon.a (mon.0) 1083 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:18.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:18 smithi169 bash[25083]: audit 2023-12-16T07:45:17.847305+0000 mon.a (mon.0) 1084 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:18.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:18 smithi169 bash[25083]: audit 2023-12-16T07:45:18.183170+0000 mgr.y (mgr.15429) 48 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:18.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:18 smithi169 bash[25083]: cluster 2023-12-16T07:45:18.710347+0000 mgr.y (mgr.15429) 49 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2023-12-16T07:45:19.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:18 smithi007 bash[25009]: audit 2023-12-16T07:45:17.837446+0000 mon.a (mon.0) 1083 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:19.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:18 smithi007 bash[25009]: audit 2023-12-16T07:45:17.847305+0000 mon.a (mon.0) 1084 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:19.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:18 smithi007 bash[25009]: audit 2023-12-16T07:45:18.183170+0000 mgr.y (mgr.15429) 48 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:19.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:18 smithi007 bash[25009]: cluster 2023-12-16T07:45:18.710347+0000 mgr.y (mgr.15429) 49 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2023-12-16T07:45:19.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:18 smithi007 bash[30436]: audit 2023-12-16T07:45:17.837446+0000 mon.a (mon.0) 1083 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:19.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:18 smithi007 bash[30436]: audit 2023-12-16T07:45:17.847305+0000 mon.a (mon.0) 1084 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:19.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:18 smithi007 bash[30436]: audit 2023-12-16T07:45:18.183170+0000 mgr.y (mgr.15429) 48 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:19.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:18 smithi007 bash[30436]: cluster 2023-12-16T07:45:18.710347+0000 mgr.y (mgr.15429) 49 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2023-12-16T07:45:19.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:18 smithi169 bash[49542]: debug 2023-12-16T07:45:18.942+0000 7f4b08f3d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T07:45:20.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:19 smithi169 bash[49542]: debug 2023-12-16T07:45:19.846+0000 7f4b08f3d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-16T07:45:20.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:19 smithi169 bash[49542]: debug 2023-12-16T07:45:19.946+0000 7f4b08f3d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T07:45:20.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:20 smithi169 bash[49542]: debug 2023-12-16T07:45:20.046+0000 7f4b08f3d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T07:45:20.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:20 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:45:20] "GET /metrics HTTP/1.1" 200 38140 "" "Prometheus/2.33.4" 2023-12-16T07:45:20.672 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:20 smithi169 bash[49542]: debug 2023-12-16T07:45:20.314+0000 7f4b08f3d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:45:20.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:20 smithi169 bash[49542]: debug 2023-12-16T07:45:20.666+0000 7f4b08f3d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T07:45:20.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:20 smithi169 bash[49542]: debug 2023-12-16T07:45:20.826+0000 7f4b08f3d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T07:45:21.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:21 smithi169 bash[49542]: debug 2023-12-16T07:45:21.042+0000 7f4b08f3d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T07:45:22.062 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:21 smithi169 bash[25083]: cluster 2023-12-16T07:45:20.711976+0000 mgr.y (mgr.15429) 50 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-12-16T07:45:22.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:21 smithi007 bash[25009]: cluster 2023-12-16T07:45:20.711976+0000 mgr.y (mgr.15429) 50 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-12-16T07:45:22.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:21 smithi007 bash[30436]: cluster 2023-12-16T07:45:20.711976+0000 mgr.y (mgr.15429) 50 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2023-12-16T07:45:22.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:22 smithi169 bash[49542]: debug 2023-12-16T07:45:22.054+0000 7f4b08f3d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T07:45:22.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:22 smithi169 bash[49542]: debug 2023-12-16T07:45:22.166+0000 7f4b08f3d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T07:45:22.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:22 smithi169 bash[49542]: debug 2023-12-16T07:45:22.286+0000 7f4b08f3d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T07:45:23.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:22 smithi169 bash[49542]: debug 2023-12-16T07:45:22.786+0000 7f4b08f3d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T07:45:23.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:23 smithi169 bash[49542]: debug 2023-12-16T07:45:23.030+0000 7f4b08f3d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T07:45:23.549 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:23 smithi169 bash[25083]: cluster 2023-12-16T07:45:22.712732+0000 mgr.y (mgr.15429) 51 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:23.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:23 smithi007 bash[30436]: cluster 2023-12-16T07:45:22.712732+0000 mgr.y (mgr.15429) 51 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:23.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:23 smithi007 bash[25009]: cluster 2023-12-16T07:45:22.712732+0000 mgr.y (mgr.15429) 51 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:23.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:23 smithi169 bash[49542]: debug 2023-12-16T07:45:23.546+0000 7f4b08f3d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T07:45:24.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:24 smithi169 bash[49542]: debug 2023-12-16T07:45:24.146+0000 7f4b08f3d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:45:25.165 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:24 smithi169 bash[49542]: debug 2023-12-16T07:45:24.710+0000 7f4b08f3d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T07:45:25.165 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:24 smithi169 bash[49542]: debug 2023-12-16T07:45:24.822+0000 7f4b08f3d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T07:45:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:25 smithi169 bash[25083]: audit 2023-12-16T07:45:24.164240+0000 mon.a (mon.0) 1085 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:25 smithi169 bash[25083]: audit 2023-12-16T07:45:24.174576+0000 mon.a (mon.0) 1086 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:25 smithi169 bash[25083]: audit 2023-12-16T07:45:24.473084+0000 mon.a (mon.0) 1087 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:25 smithi169 bash[25083]: audit 2023-12-16T07:45:24.480680+0000 mon.a (mon.0) 1088 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:25 smithi169 bash[25083]: cluster 2023-12-16T07:45:24.714382+0000 mgr.y (mgr.15429) 52 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:45:25.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:25 smithi169 bash[49542]: debug 2023-12-16T07:45:25.186+0000 7f4b08f3d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T07:45:25.434 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:25 smithi169 bash[49542]: debug 2023-12-16T07:45:25.302+0000 7f4b08f3d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T07:45:25.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:25 smithi007 bash[30436]: audit 2023-12-16T07:45:24.164240+0000 mon.a (mon.0) 1085 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:25.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:25 smithi007 bash[30436]: audit 2023-12-16T07:45:24.174576+0000 mon.a (mon.0) 1086 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:25.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:25 smithi007 bash[30436]: audit 2023-12-16T07:45:24.473084+0000 mon.a (mon.0) 1087 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:25.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:25 smithi007 bash[30436]: audit 2023-12-16T07:45:24.480680+0000 mon.a (mon.0) 1088 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:25.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:25 smithi007 bash[30436]: cluster 2023-12-16T07:45:24.714382+0000 mgr.y (mgr.15429) 52 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:45:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:25 smithi007 bash[25009]: audit 2023-12-16T07:45:24.164240+0000 mon.a (mon.0) 1085 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:25 smithi007 bash[25009]: audit 2023-12-16T07:45:24.174576+0000 mon.a (mon.0) 1086 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:25 smithi007 bash[25009]: audit 2023-12-16T07:45:24.473084+0000 mon.a (mon.0) 1087 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:25 smithi007 bash[25009]: audit 2023-12-16T07:45:24.480680+0000 mon.a (mon.0) 1088 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:25 smithi007 bash[25009]: cluster 2023-12-16T07:45:24.714382+0000 mgr.y (mgr.15429) 52 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:45:25.584 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:45:25.885 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:25 smithi169 bash[49542]: debug 2023-12-16T07:45:25.450+0000 7f4b08f3d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T07:45:25.885 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:25 smithi169 bash[49542]: debug 2023-12-16T07:45:25.570+0000 7f4b08f3d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T07:45:25.886 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:25 smithi169 bash[49542]: debug 2023-12-16T07:45:25.694+0000 7f4b08f3d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T07:45:26.184 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:25 smithi169 bash[49542]: debug 2023-12-16T07:45:25.882+0000 7f4b08f3d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T07:45:26.184 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:26 smithi169 bash[49542]: debug 2023-12-16T07:45:26.010+0000 7f4b08f3d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T07:45:26.354 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:45:26.354 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (114s) 1s ago 9m 16.1M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:45:26.354 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (69s) 2s ago 8m 43.2M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:45:26.354 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (20s) 1s ago 6m 44.0M - 3.5 e1d6a67b021e 8d8009d0c83f 2023-12-16T07:45:26.354 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283,8765 running (8s) 2s ago 15m 94.7M - 19.0.0-191-g6d933094 a29c258b5d6f 5295ba35c2cd 2023-12-16T07:45:26.354 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (62s) 1s ago 17m 483M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:45:26.354 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (17m) 1s ago 17m 61.0M 2048M 17.2.0 e1d6a67b021e c0468ecbd44e 2023-12-16T07:45:26.354 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (15m) 2s ago 15m 60.5M 2048M 17.2.0 e1d6a67b021e b47a960c6102 2023-12-16T07:45:26.354 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (15m) 1s ago 15m 55.5M 2048M 17.2.0 e1d6a67b021e c659b986f201 2023-12-16T07:45:26.354 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (2m) 1s ago 9m 10.7M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:45:26.354 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (93s) 2s ago 9m 11.2M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:45:26.355 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (14m) 1s ago 14m 57.8M 2782M 17.2.0 e1d6a67b021e dbc4d9538238 2023-12-16T07:45:26.355 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (13m) 1s ago 13m 58.4M 2782M 17.2.0 e1d6a67b021e 14637e0f389c 2023-12-16T07:45:26.355 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (13m) 1s ago 13m 56.8M 2782M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:45:26.355 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (12m) 1s ago 12m 56.5M 2782M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:45:26.355 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (12m) 2s ago 12m 57.9M 3037M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:45:26.355 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (11m) 2s ago 11m 54.5M 3037M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:45:26.355 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (10m) 2s ago 10m 54.9M 3037M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:45:26.355 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (10m) 2s ago 10m 55.7M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:45:26.355 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (14s) 2s ago 9m 47.9M - 2.33.4 514e6a882f6e 2652ce1b8540 2023-12-16T07:45:26.355 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (7m) 1s ago 7m 86.5M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:45:26.355 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (7m) 2s ago 7m 86.4M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:45:26.355 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (6m) 1s ago 6m 86.1M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:45:26.355 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (6m) 2s ago 6m 86.3M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:45:26.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:26 smithi169 bash[25083]: audit 2023-12-16T07:45:25.180473+0000 mon.a (mon.0) 1089 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:26.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:26 smithi169 bash[25083]: audit 2023-12-16T07:45:25.186804+0000 mon.a (mon.0) 1090 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:26.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:26 smithi169 bash[25083]: audit 2023-12-16T07:45:25.565919+0000 mgr.y (mgr.15429) 53 : audit [DBG] from='client.25435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:26.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:26 smithi007 bash[30436]: audit 2023-12-16T07:45:25.180473+0000 mon.a (mon.0) 1089 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:26.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:26 smithi007 bash[30436]: audit 2023-12-16T07:45:25.186804+0000 mon.a (mon.0) 1090 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:26.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:26 smithi007 bash[30436]: audit 2023-12-16T07:45:25.565919+0000 mgr.y (mgr.15429) 53 : audit [DBG] from='client.25435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:26.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:26 smithi007 bash[25009]: audit 2023-12-16T07:45:25.180473+0000 mon.a (mon.0) 1089 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:26.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:26 smithi007 bash[25009]: audit 2023-12-16T07:45:25.186804+0000 mon.a (mon.0) 1090 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:26.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:26 smithi007 bash[25009]: audit 2023-12-16T07:45:25.565919+0000 mgr.y (mgr.15429) 53 : audit [DBG] from='client.25435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:26.802 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:45:26.802 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:45:26.802 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2023-12-16T07:45:26.802 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:45:26.802 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:45:26.802 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:45:26.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:45:26.803 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:45:26.803 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2023-12-16T07:45:26.803 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:45:26.803 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2023-12-16T07:45:26.803 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:45:26.804 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:45:26.804 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:45:26.804 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:45:26.804 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2023-12-16T07:45:26.804 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:45:26.804 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:45:26.805 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:45:27.191 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:45:27.191 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:45:27.191 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:45:27.191 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:45:27.191 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2023-12-16T07:45:27.191 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2023-12-16T07:45:27.192 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:45:27.192 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "2/23 daemons upgraded", 2023-12-16T07:45:27.192 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading mgr daemons", 2023-12-16T07:45:27.192 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:45:27.192 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:45:27.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:27 smithi007 bash[30436]: audit 2023-12-16T07:45:25.953941+0000 mgr.y (mgr.15429) 54 : audit [DBG] from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:27.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:27 smithi007 bash[30436]: audit 2023-12-16T07:45:26.328500+0000 mgr.y (mgr.15429) 55 : audit [DBG] from='client.15543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:27.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:27 smithi007 bash[30436]: cluster 2023-12-16T07:45:26.715126+0000 mgr.y (mgr.15429) 56 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:27.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:27 smithi007 bash[30436]: audit 2023-12-16T07:45:26.799016+0000 mon.a (mon.0) 1091 : audit [DBG] from='client.? 172.21.15.7:0/466101067' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:45:27.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:27 smithi007 bash[25009]: audit 2023-12-16T07:45:25.953941+0000 mgr.y (mgr.15429) 54 : audit [DBG] from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:27.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:27 smithi007 bash[25009]: audit 2023-12-16T07:45:26.328500+0000 mgr.y (mgr.15429) 55 : audit [DBG] from='client.15543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:27.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:27 smithi007 bash[25009]: cluster 2023-12-16T07:45:26.715126+0000 mgr.y (mgr.15429) 56 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:27.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:27 smithi007 bash[25009]: audit 2023-12-16T07:45:26.799016+0000 mon.a (mon.0) 1091 : audit [DBG] from='client.? 172.21.15.7:0/466101067' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:45:27.633 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:45:27.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:27 smithi169 bash[25083]: audit 2023-12-16T07:45:25.953941+0000 mgr.y (mgr.15429) 54 : audit [DBG] from='client.15537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:27.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:27 smithi169 bash[25083]: audit 2023-12-16T07:45:26.328500+0000 mgr.y (mgr.15429) 55 : audit [DBG] from='client.15543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:27.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:27 smithi169 bash[25083]: cluster 2023-12-16T07:45:26.715126+0000 mgr.y (mgr.15429) 56 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:27.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:27 smithi169 bash[25083]: audit 2023-12-16T07:45:26.799016+0000 mon.a (mon.0) 1091 : audit [DBG] from='client.? 172.21.15.7:0/466101067' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:45:27.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:27 smithi169 bash[49542]: debug 2023-12-16T07:45:27.242+0000 7f4b08f3d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T07:45:27.684 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:27 smithi169 bash[49542]: [16/Dec/2023:07:45:27] ENGINE Bus STARTING 2023-12-16T07:45:27.684 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:27 smithi169 bash[49542]: CherryPy Checker: 2023-12-16T07:45:27.684 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:27 smithi169 bash[49542]: The Application mounted at '' has an empty config. 2023-12-16T07:45:27.684 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:27 smithi169 bash[49542]: [16/Dec/2023:07:45:27] ENGINE Serving on http://:::9283 2023-12-16T07:45:27.684 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:27 smithi169 bash[49542]: [16/Dec/2023:07:45:27] ENGINE Bus STARTED 2023-12-16T07:45:28.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[25009]: audit 2023-12-16T07:45:27.188729+0000 mgr.y (mgr.15429) 57 : audit [DBG] from='client.25459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:28.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[25009]: cluster 2023-12-16T07:45:27.264110+0000 mon.a (mon.0) 1092 : cluster [DBG] Standby manager daemon x restarted 2023-12-16T07:45:28.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[25009]: cluster 2023-12-16T07:45:27.264325+0000 mon.a (mon.0) 1093 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:45:28.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[25009]: audit 2023-12-16T07:45:27.269346+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.? 172.21.15.169:0/1442914838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:45:28.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[25009]: audit 2023-12-16T07:45:27.270769+0000 mon.b (mon.2) 91 : audit [DBG] from='mgr.? 172.21.15.169:0/1442914838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:45:28.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[25009]: audit 2023-12-16T07:45:27.276499+0000 mon.b (mon.2) 92 : audit [DBG] from='mgr.? 172.21.15.169:0/1442914838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:45:28.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[25009]: audit 2023-12-16T07:45:27.277314+0000 mon.b (mon.2) 93 : audit [DBG] from='mgr.? 172.21.15.169:0/1442914838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:45:28.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[25009]: audit 2023-12-16T07:45:27.631513+0000 mon.a (mon.0) 1094 : audit [DBG] from='client.? 172.21.15.7:0/4085484067' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:45:28.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[25009]: audit 2023-12-16T07:45:27.853406+0000 mon.a (mon.0) 1095 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:45:28.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[30436]: audit 2023-12-16T07:45:27.188729+0000 mgr.y (mgr.15429) 57 : audit [DBG] from='client.25459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:28.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[30436]: cluster 2023-12-16T07:45:27.264110+0000 mon.a (mon.0) 1092 : cluster [DBG] Standby manager daemon x restarted 2023-12-16T07:45:28.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[30436]: cluster 2023-12-16T07:45:27.264325+0000 mon.a (mon.0) 1093 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:45:28.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[30436]: audit 2023-12-16T07:45:27.269346+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.? 172.21.15.169:0/1442914838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:45:28.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[30436]: audit 2023-12-16T07:45:27.270769+0000 mon.b (mon.2) 91 : audit [DBG] from='mgr.? 172.21.15.169:0/1442914838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:45:28.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[30436]: audit 2023-12-16T07:45:27.276499+0000 mon.b (mon.2) 92 : audit [DBG] from='mgr.? 172.21.15.169:0/1442914838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:45:28.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[30436]: audit 2023-12-16T07:45:27.277314+0000 mon.b (mon.2) 93 : audit [DBG] from='mgr.? 172.21.15.169:0/1442914838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:45:28.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[30436]: audit 2023-12-16T07:45:27.631513+0000 mon.a (mon.0) 1094 : audit [DBG] from='client.? 172.21.15.7:0/4085484067' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:45:28.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:28 smithi007 bash[30436]: audit 2023-12-16T07:45:27.853406+0000 mon.a (mon.0) 1095 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:45:28.663 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:28 smithi169 bash[25083]: audit 2023-12-16T07:45:27.188729+0000 mgr.y (mgr.15429) 57 : audit [DBG] from='client.25459 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:28.663 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:28 smithi169 bash[25083]: cluster 2023-12-16T07:45:27.264110+0000 mon.a (mon.0) 1092 : cluster [DBG] Standby manager daemon x restarted 2023-12-16T07:45:28.663 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:28 smithi169 bash[25083]: cluster 2023-12-16T07:45:27.264325+0000 mon.a (mon.0) 1093 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:45:28.663 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:28 smithi169 bash[25083]: audit 2023-12-16T07:45:27.269346+0000 mon.b (mon.2) 90 : audit [DBG] from='mgr.? 172.21.15.169:0/1442914838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:45:28.663 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:28 smithi169 bash[25083]: audit 2023-12-16T07:45:27.270769+0000 mon.b (mon.2) 91 : audit [DBG] from='mgr.? 172.21.15.169:0/1442914838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:45:28.663 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:28 smithi169 bash[25083]: audit 2023-12-16T07:45:27.276499+0000 mon.b (mon.2) 92 : audit [DBG] from='mgr.? 172.21.15.169:0/1442914838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:45:28.664 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:28 smithi169 bash[25083]: audit 2023-12-16T07:45:27.277314+0000 mon.b (mon.2) 93 : audit [DBG] from='mgr.? 172.21.15.169:0/1442914838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:45:28.664 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:28 smithi169 bash[25083]: audit 2023-12-16T07:45:27.631513+0000 mon.a (mon.0) 1094 : audit [DBG] from='client.? 172.21.15.7:0/4085484067' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:45:28.664 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:28 smithi169 bash[25083]: audit 2023-12-16T07:45:27.853406+0000 mon.a (mon.0) 1095 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:45:28.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:28 smithi169 bash[49087]: ts=2023-12-16T07:45:28.662Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:45:29.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:29 smithi007 bash[30436]: audit 2023-12-16T07:45:28.193621+0000 mgr.y (mgr.15429) 58 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:29.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:29 smithi007 bash[30436]: cluster 2023-12-16T07:45:28.211831+0000 mon.a (mon.0) 1096 : cluster [DBG] mgrmap e33: y(active, since 46s), standbys: x 2023-12-16T07:45:29.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:29 smithi007 bash[30436]: cluster 2023-12-16T07:45:28.716153+0000 mgr.y (mgr.15429) 59 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:29.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:29 smithi007 bash[25009]: audit 2023-12-16T07:45:28.193621+0000 mgr.y (mgr.15429) 58 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:29.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:29 smithi007 bash[25009]: cluster 2023-12-16T07:45:28.211831+0000 mon.a (mon.0) 1096 : cluster [DBG] mgrmap e33: y(active, since 46s), standbys: x 2023-12-16T07:45:29.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:29 smithi007 bash[25009]: cluster 2023-12-16T07:45:28.716153+0000 mgr.y (mgr.15429) 59 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:29.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:29 smithi169 bash[25083]: audit 2023-12-16T07:45:28.193621+0000 mgr.y (mgr.15429) 58 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:29.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:29 smithi169 bash[25083]: cluster 2023-12-16T07:45:28.211831+0000 mon.a (mon.0) 1096 : cluster [DBG] mgrmap e33: y(active, since 46s), standbys: x 2023-12-16T07:45:29.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:29 smithi169 bash[25083]: cluster 2023-12-16T07:45:28.716153+0000 mgr.y (mgr.15429) 59 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:30.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:30 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:45:30] "GET /metrics HTTP/1.1" 200 38141 "" "Prometheus/2.33.4" 2023-12-16T07:45:31.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:31 smithi169 bash[25083]: cluster 2023-12-16T07:45:30.717904+0000 mgr.y (mgr.15429) 60 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:45:32.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:31 smithi007 bash[30436]: cluster 2023-12-16T07:45:30.717904+0000 mgr.y (mgr.15429) 60 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:45:32.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:31 smithi007 bash[25009]: cluster 2023-12-16T07:45:30.717904+0000 mgr.y (mgr.15429) 60 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:45:33.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:33 smithi007 bash[25009]: cluster 2023-12-16T07:45:32.718872+0000 mgr.y (mgr.15429) 61 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:33.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:33 smithi007 bash[30436]: cluster 2023-12-16T07:45:32.718872+0000 mgr.y (mgr.15429) 61 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:33.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:33 smithi169 bash[25083]: cluster 2023-12-16T07:45:32.718872+0000 mgr.y (mgr.15429) 61 : cluster [DBG] pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:36.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: cluster 2023-12-16T07:45:34.720730+0000 mgr.y (mgr.15429) 62 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:45:36.671 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: audit 2023-12-16T07:45:35.039804+0000 mon.a (mon.0) 1097 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:36.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: audit 2023-12-16T07:45:35.049543+0000 mon.a (mon.0) 1098 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:36.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: audit 2023-12-16T07:45:35.051195+0000 mon.a (mon.0) 1099 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:36.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: audit 2023-12-16T07:45:35.052886+0000 mon.a (mon.0) 1100 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:45:36.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: audit 2023-12-16T07:45:35.061586+0000 mon.a (mon.0) 1101 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:36.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: audit 2023-12-16T07:45:35.151174+0000 mon.a (mon.0) 1102 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:45:36.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: audit 2023-12-16T07:45:35.155052+0000 mon.a (mon.0) 1103 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:45:36.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: audit 2023-12-16T07:45:35.168411+0000 mon.a (mon.0) 1104 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:36.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: audit 2023-12-16T07:45:35.170219+0000 mon.a (mon.0) 1105 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2023-12-16T07:45:36.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: audit 2023-12-16T07:45:35.176880+0000 mon.a (mon.0) 1106 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2023-12-16T07:45:36.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: audit 2023-12-16T07:45:35.178829+0000 mon.a (mon.0) 1107 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2023-12-16T07:45:36.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: audit 2023-12-16T07:45:35.186400+0000 mon.a (mon.0) 1108 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2023-12-16T07:45:36.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: audit 2023-12-16T07:45:35.188930+0000 mon.a (mon.0) 1109 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-12-16T07:45:36.673 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[30436]: audit 2023-12-16T07:45:35.190409+0000 mon.a (mon.0) 1110 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2023-12-16T07:45:36.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: cluster 2023-12-16T07:45:34.720730+0000 mgr.y (mgr.15429) 62 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:45:36.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: audit 2023-12-16T07:45:35.039804+0000 mon.a (mon.0) 1097 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:36.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: audit 2023-12-16T07:45:35.049543+0000 mon.a (mon.0) 1098 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:36.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: audit 2023-12-16T07:45:35.051195+0000 mon.a (mon.0) 1099 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:36.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: audit 2023-12-16T07:45:35.052886+0000 mon.a (mon.0) 1100 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:45:36.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: audit 2023-12-16T07:45:35.061586+0000 mon.a (mon.0) 1101 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:36.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: audit 2023-12-16T07:45:35.151174+0000 mon.a (mon.0) 1102 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:45:36.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: audit 2023-12-16T07:45:35.155052+0000 mon.a (mon.0) 1103 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:45:36.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: audit 2023-12-16T07:45:35.168411+0000 mon.a (mon.0) 1104 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: audit 2023-12-16T07:45:35.170219+0000 mon.a (mon.0) 1105 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2023-12-16T07:45:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: audit 2023-12-16T07:45:35.176880+0000 mon.a (mon.0) 1106 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2023-12-16T07:45:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: audit 2023-12-16T07:45:35.178829+0000 mon.a (mon.0) 1107 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2023-12-16T07:45:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: audit 2023-12-16T07:45:35.186400+0000 mon.a (mon.0) 1108 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2023-12-16T07:45:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: audit 2023-12-16T07:45:35.188930+0000 mon.a (mon.0) 1109 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-12-16T07:45:36.674 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:35 smithi007 bash[25009]: audit 2023-12-16T07:45:35.190409+0000 mon.a (mon.0) 1110 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2023-12-16T07:45:36.674 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: cluster 2023-12-16T07:45:34.720730+0000 mgr.y (mgr.15429) 62 : cluster [DBG] pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:45:36.674 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: audit 2023-12-16T07:45:35.039804+0000 mon.a (mon.0) 1097 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:36.675 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: audit 2023-12-16T07:45:35.049543+0000 mon.a (mon.0) 1098 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:36.675 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: audit 2023-12-16T07:45:35.051195+0000 mon.a (mon.0) 1099 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:36.675 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: audit 2023-12-16T07:45:35.052886+0000 mon.a (mon.0) 1100 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:45:36.675 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: audit 2023-12-16T07:45:35.061586+0000 mon.a (mon.0) 1101 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:36.675 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: audit 2023-12-16T07:45:35.151174+0000 mon.a (mon.0) 1102 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:45:36.675 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: audit 2023-12-16T07:45:35.155052+0000 mon.a (mon.0) 1103 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:45:36.675 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: audit 2023-12-16T07:45:35.168411+0000 mon.a (mon.0) 1104 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:36.675 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: audit 2023-12-16T07:45:35.170219+0000 mon.a (mon.0) 1105 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2023-12-16T07:45:36.675 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: audit 2023-12-16T07:45:35.176880+0000 mon.a (mon.0) 1106 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2023-12-16T07:45:36.675 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: audit 2023-12-16T07:45:35.178829+0000 mon.a (mon.0) 1107 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2023-12-16T07:45:36.675 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: audit 2023-12-16T07:45:35.186400+0000 mon.a (mon.0) 1108 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2023-12-16T07:45:36.675 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: audit 2023-12-16T07:45:35.188930+0000 mon.a (mon.0) 1109 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-12-16T07:45:36.675 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:35 smithi169 bash[25083]: audit 2023-12-16T07:45:35.190409+0000 mon.a (mon.0) 1110 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2023-12-16T07:45:37.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:36 smithi007 bash[25009]: cephadm 2023-12-16T07:45:35.156452+0000 mgr.y (mgr.15429) 63 : cephadm [INF] Upgrade: Setting container_image for all mgr 2023-12-16T07:45:37.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:36 smithi007 bash[25009]: cephadm 2023-12-16T07:45:35.191625+0000 mgr.y (mgr.15429) 64 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2023-12-16T07:45:37.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:36 smithi007 bash[30436]: cephadm 2023-12-16T07:45:35.156452+0000 mgr.y (mgr.15429) 63 : cephadm [INF] Upgrade: Setting container_image for all mgr 2023-12-16T07:45:37.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:36 smithi007 bash[30436]: cephadm 2023-12-16T07:45:35.191625+0000 mgr.y (mgr.15429) 64 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2023-12-16T07:45:37.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:36 smithi169 bash[25083]: cephadm 2023-12-16T07:45:35.156452+0000 mgr.y (mgr.15429) 63 : cephadm [INF] Upgrade: Setting container_image for all mgr 2023-12-16T07:45:37.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:36 smithi169 bash[25083]: cephadm 2023-12-16T07:45:35.191625+0000 mgr.y (mgr.15429) 64 : cephadm [INF] Upgrade: It appears safe to stop mon.a 2023-12-16T07:45:38.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:37 smithi007 bash[25009]: cluster 2023-12-16T07:45:36.721555+0000 mgr.y (mgr.15429) 65 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:38.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:37 smithi007 bash[30436]: cluster 2023-12-16T07:45:36.721555+0000 mgr.y (mgr.15429) 65 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:38.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:37 smithi169 bash[25083]: cluster 2023-12-16T07:45:36.721555+0000 mgr.y (mgr.15429) 65 : cluster [DBG] pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:38.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:38 smithi169 bash[49087]: ts=2023-12-16T07:45:38.662Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:45:39.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:39 smithi169 bash[25083]: cephadm 2023-12-16T07:45:38.163453+0000 mgr.y (mgr.15429) 66 : cephadm [INF] Upgrade: Updating mon.a 2023-12-16T07:45:39.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:39 smithi169 bash[25083]: audit 2023-12-16T07:45:38.172715+0000 mon.a (mon.0) 1111 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:39.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:39 smithi169 bash[25083]: audit 2023-12-16T07:45:38.174790+0000 mon.a (mon.0) 1112 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:45:39.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:39 smithi169 bash[25083]: audit 2023-12-16T07:45:38.176142+0000 mon.a (mon.0) 1113 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:45:39.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:39 smithi169 bash[25083]: audit 2023-12-16T07:45:38.177477+0000 mon.a (mon.0) 1114 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:39.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:39 smithi169 bash[25083]: cephadm 2023-12-16T07:45:38.179416+0000 mgr.y (mgr.15429) 67 : cephadm [INF] Deploying daemon mon.a on smithi007 2023-12-16T07:45:39.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:39 smithi169 bash[25083]: audit 2023-12-16T07:45:38.203023+0000 mgr.y (mgr.15429) 68 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:39.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:39 smithi169 bash[25083]: cluster 2023-12-16T07:45:38.722484+0000 mgr.y (mgr.15429) 69 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:39.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[25009]: cephadm 2023-12-16T07:45:38.163453+0000 mgr.y (mgr.15429) 66 : cephadm [INF] Upgrade: Updating mon.a 2023-12-16T07:45:39.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[25009]: audit 2023-12-16T07:45:38.172715+0000 mon.a (mon.0) 1111 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:39.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[25009]: audit 2023-12-16T07:45:38.174790+0000 mon.a (mon.0) 1112 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:45:39.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[25009]: audit 2023-12-16T07:45:38.176142+0000 mon.a (mon.0) 1113 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:45:39.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[25009]: audit 2023-12-16T07:45:38.177477+0000 mon.a (mon.0) 1114 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:39.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[25009]: cephadm 2023-12-16T07:45:38.179416+0000 mgr.y (mgr.15429) 67 : cephadm [INF] Deploying daemon mon.a on smithi007 2023-12-16T07:45:39.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[25009]: audit 2023-12-16T07:45:38.203023+0000 mgr.y (mgr.15429) 68 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:39.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[25009]: cluster 2023-12-16T07:45:38.722484+0000 mgr.y (mgr.15429) 69 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:39.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[30436]: cephadm 2023-12-16T07:45:38.163453+0000 mgr.y (mgr.15429) 66 : cephadm [INF] Upgrade: Updating mon.a 2023-12-16T07:45:39.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[30436]: audit 2023-12-16T07:45:38.172715+0000 mon.a (mon.0) 1111 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' 2023-12-16T07:45:39.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[30436]: audit 2023-12-16T07:45:38.174790+0000 mon.a (mon.0) 1112 : audit [INF] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:45:39.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[30436]: audit 2023-12-16T07:45:38.176142+0000 mon.a (mon.0) 1113 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:45:39.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[30436]: audit 2023-12-16T07:45:38.177477+0000 mon.a (mon.0) 1114 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:45:39.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[30436]: cephadm 2023-12-16T07:45:38.179416+0000 mgr.y (mgr.15429) 67 : cephadm [INF] Deploying daemon mon.a on smithi007 2023-12-16T07:45:39.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[30436]: audit 2023-12-16T07:45:38.203023+0000 mgr.y (mgr.15429) 68 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:39.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:39 smithi007 bash[30436]: cluster 2023-12-16T07:45:38.722484+0000 mgr.y (mgr.15429) 69 : cluster [DBG] pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:40.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:40 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:45:40] "GET /metrics HTTP/1.1" 200 38141 "" "Prometheus/2.33.4" 2023-12-16T07:45:41.415 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:41 smithi007 bash[61260]: [16/Dec/2023:07:45:41] ENGINE Bus STOPPING 2023-12-16T07:45:41.415 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:41 smithi007 bash[61260]: [16/Dec/2023:07:45:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2023-12-16T07:45:41.415 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:41 smithi007 bash[61260]: [16/Dec/2023:07:45:41] ENGINE Bus STOPPED 2023-12-16T07:45:41.415 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:41 smithi007 bash[61260]: [16/Dec/2023:07:45:41] ENGINE Bus STARTING 2023-12-16T07:45:41.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:41 smithi007 systemd[1]: Stopping Ceph mon.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:45:41.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:41 smithi007 bash[25009]: debug 2023-12-16T07:45:41.276+0000 7fdca2f72700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-16T07:45:41.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:41 smithi007 bash[25009]: debug 2023-12-16T07:45:41.276+0000 7fdca2f72700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-12-16T07:45:41.708 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:41 smithi007 bash[61260]: [16/Dec/2023:07:45:41] ENGINE Serving on http://:::9283 2023-12-16T07:45:41.708 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:41 smithi007 bash[61260]: [16/Dec/2023:07:45:41] ENGINE Bus STARTED 2023-12-16T07:45:42.371 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:42 smithi007 bash[64332]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mon-a 2023-12-16T07:45:42.685 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:42 smithi007 bash[64413]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mon-a 2023-12-16T07:45:42.685 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:42 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.a.service: Succeeded. 2023-12-16T07:45:42.685 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:42 smithi007 systemd[1]: Stopped Ceph mon.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:45:42.685 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:42 smithi007 systemd[1]: Started Ceph mon.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:45:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.719+0000 7f4f68746c80 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-16T07:45:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.719+0000 7f4f68746c80 0 ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev), process ceph-mon, pid 7 2023-12-16T07:45:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.719+0000 7f4f68746c80 0 pidfile_write: ignore empty --pid-file 2023-12-16T07:45:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 0 load: jerasure load: lrc 2023-12-16T07:45:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: RocksDB version: 7.9.2 2023-12-16T07:45:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Git sha 0 2023-12-16T07:45:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Compile date 2023-12-15 16:32:04 2023-12-16T07:45:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: DB SUMMARY 2023-12-16T07:45:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: DB Session ID: 9A6KOWYETGATBZGNXD8K 2023-12-16T07:45:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: CURRENT file: CURRENT 2023-12-16T07:45:44.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: IDENTITY file: IDENTITY 2023-12-16T07:45:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: MANIFEST file: MANIFEST-000015 size: 1338 Bytes 2023-12-16T07:45:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000036.sst 2023-12-16T07:45:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000034.log size: 452341 ; 2023-12-16T07:45:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.error_if_exists: 0 2023-12-16T07:45:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.create_if_missing: 0 2023-12-16T07:45:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.paranoid_checks: 1 2023-12-16T07:45:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2023-12-16T07:45:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-12-16T07:45:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-12-16T07:45:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.env: 0x56293c2b7a60 2023-12-16T07:45:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.fs: PosixFileSystem 2023-12-16T07:45:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.info_log: 0x56293daff7a0 2023-12-16T07:45:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-16T07:45:44.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.statistics: (nil) 2023-12-16T07:45:44.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.use_fsync: 0 2023-12-16T07:45:44.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_log_file_size: 0 2023-12-16T07:45:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-16T07:45:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-16T07:45:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-16T07:45:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-16T07:45:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.allow_fallocate: 1 2023-12-16T07:45:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-16T07:45:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-16T07:45:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.use_direct_reads: 0 2023-12-16T07:45:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-16T07:45:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.create_missing_column_families: 0 2023-12-16T07:45:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.db_log_dir: 2023-12-16T07:45:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.wal_dir: 2023-12-16T07:45:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-16T07:45:44.071 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.advise_random_on_open: 1 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.write_buffer_manager: 0x56293dad32c0 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.rate_limiter: (nil) 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-16T07:45:44.072 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-16T07:45:44.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.unordered_write: 0 2023-12-16T07:45:44.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-16T07:45:44.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-16T07:45:44.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-16T07:45:44.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-16T07:45:44.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.row_cache: None 2023-12-16T07:45:44.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.wal_filter: None 2023-12-16T07:45:44.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-16T07:45:44.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-16T07:45:44.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.two_write_queues: 0 2023-12-16T07:45:44.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.manual_wal_flush: 0 2023-12-16T07:45:44.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.wal_compression: 0 2023-12-16T07:45:44.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.atomic_flush: 0 2023-12-16T07:45:44.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-16T07:45:44.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-16T07:45:44.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-16T07:45:44.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.log_readahead_size: 0 2023-12-16T07:45:44.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-12-16T07:45:44.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.best_efforts_recovery: 0 2023-12-16T07:45:44.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-12-16T07:45:44.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-12-16T07:45:44.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.allow_data_in_errors: 0 2023-12-16T07:45:44.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.db_host_id: __hostname__ 2023-12-16T07:45:44.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.enforce_single_del_contracts: true 2023-12-16T07:45:44.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_background_jobs: 2 2023-12-16T07:45:44.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_background_compactions: -1 2023-12-16T07:45:44.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_subcompactions: 1 2023-12-16T07:45:44.074 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_total_wal_size: 0 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_open_files: -1 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.bytes_per_sync: 0 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_background_flushes: -1 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Compression algorithms supported: 2023-12-16T07:45:44.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-16T07:45:44.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: kZSTD supported: 0 2023-12-16T07:45:44.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: kXpressCompression supported: 0 2023-12-16T07:45:44.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-16T07:45:44.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: kZlibCompression supported: 1 2023-12-16T07:45:44.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: kSnappyCompression supported: 1 2023-12-16T07:45:44.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: kLZ4Compression supported: 1 2023-12-16T07:45:44.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: kBZip2Compression supported: 0 2023-12-16T07:45:44.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-16T07:45:44.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2023-12-16T07:45:44.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2023-12-16T07:45:44.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-12-16T07:45:44.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-16T07:45:44.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.merge_operator: 2023-12-16T07:45:44.076 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compaction_filter: None 2023-12-16T07:45:44.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compaction_filter_factory: None 2023-12-16T07:45:44.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.sst_partitioner_factory: None 2023-12-16T07:45:44.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-16T07:45:44.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-16T07:45:44.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56293daff8c0) 2023-12-16T07:45:44.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: cache_index_and_filter_blocks: 1 2023-12-16T07:45:44.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-16T07:45:44.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-16T07:45:44.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: pin_top_level_index_and_filter: 1 2023-12-16T07:45:44.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: index_type: 0 2023-12-16T07:45:44.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: data_block_index_type: 0 2023-12-16T07:45:44.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: index_shortening: 1 2023-12-16T07:45:44.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: data_block_hash_table_util_ratio: 0.750000 2023-12-16T07:45:44.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: checksum: 4 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: no_block_cache: 0 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: block_cache: 0x56293db53090 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: block_cache_name: BinnedLRUCache 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: block_cache_options: 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: capacity : 536870912 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: num_shard_bits : 4 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: strict_capacity_limit : 0 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: high_pri_pool_ratio: 0.000 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: block_cache_compressed: (nil) 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: persistent_cache: (nil) 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: block_size: 4096 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: block_size_deviation: 10 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: block_restart_interval: 16 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: index_block_restart_interval: 1 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: metadata_block_size: 4096 2023-12-16T07:45:44.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: partition_filters: 0 2023-12-16T07:45:44.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: use_delta_encoding: 1 2023-12-16T07:45:44.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: filter_policy: bloomfilter 2023-12-16T07:45:44.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: whole_key_filtering: 1 2023-12-16T07:45:44.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: verify_compression: 0 2023-12-16T07:45:44.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: read_amp_bytes_per_bit: 0 2023-12-16T07:45:44.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: format_version: 5 2023-12-16T07:45:44.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: enable_index_compression: 1 2023-12-16T07:45:44.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: block_align: 0 2023-12-16T07:45:44.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: max_auto_readahead_size: 262144 2023-12-16T07:45:44.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: prepopulate_block_cache: 0 2023-12-16T07:45:44.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: initial_auto_readahead_size: 8192 2023-12-16T07:45:44.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: num_file_reads_for_auto_readahead: 2 2023-12-16T07:45:44.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-16T07:45:44.079 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compression: NoCompression 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.num_levels: 7 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-12-16T07:45:44.080 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-16T07:45:44.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-12-16T07:45:44.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-12-16T07:45:44.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-16T07:45:44.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compression_opts.level: 32767 2023-12-16T07:45:44.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-16T07:45:44.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-16T07:45:44.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-16T07:45:44.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-12-16T07:45:44.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-12-16T07:45:44.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compression_opts.enabled: false 2023-12-16T07:45:44.081 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-12-16T07:45:44.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-16T07:45:44.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-16T07:45:44.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-16T07:45:44.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-16T07:45:44.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-16T07:45:44.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-16T07:45:44.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-16T07:45:44.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-16T07:45:44.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-16T07:45:44.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-16T07:45:44.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-16T07:45:44.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-16T07:45:44.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-16T07:45:44.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-16T07:45:44.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-16T07:45:44.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-16T07:45:44.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-16T07:45:44.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-12-16T07:45:44.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.arena_block_size: 1048576 2023-12-16T07:45:44.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-16T07:45:44.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.table_properties_collectors: 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.inplace_update_support: 0 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-16T07:45:44.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-16T07:45:44.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-16T07:45:44.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-16T07:45:44.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.bloom_locality: 0 2023-12-16T07:45:44.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.max_successive_merges: 0 2023-12-16T07:45:44.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-16T07:45:44.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-16T07:45:44.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.force_consistency_checks: 1 2023-12-16T07:45:44.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-16T07:45:44.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.ttl: 2592000 2023-12-16T07:45:44.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-16T07:45:44.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2023-12-16T07:45:44.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2023-12-16T07:45:44.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.enable_blob_files: false 2023-12-16T07:45:44.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.min_blob_size: 0 2023-12-16T07:45:44.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.blob_file_size: 268435456 2023-12-16T07:45:44.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.blob_compression_type: NoCompression 2023-12-16T07:45:44.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-12-16T07:45:44.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-12-16T07:45:44.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-12-16T07:45:44.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2023-12-16T07:45:44.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.blob_file_starting_level: 0 2023-12-16T07:45:44.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-12-16T07:45:44.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f5df0e700 3 rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 36.sst 2023-12-16T07:45:44.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-12-16T07:45:44.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 38, last_sequence is 17024, log_number is 34,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-16T07:45:44.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 34 2023-12-16T07:45:44.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: efa1eb90-5256-4a90-8760-5f545d6f2ce7 2023-12-16T07:45:44.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702712743734283, "job": 1, "event": "recovery_started", "wal_files": [34]} 2023-12-16T07:45:44.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.727+0000 7f4f68746c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #34 mode 2 2023-12-16T07:45:44.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.735+0000 7f4f68746c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702712743739700, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 39, "file_size": 450031, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 17025, "largest_seqno": 17251, "table_properties": {"data_size": 448064, "index_size": 635, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 325, "raw_key_size": 3156, "raw_average_key_size": 26, "raw_value_size": 445361, "raw_average_value_size": 3680, "num_data_blocks": 28, "num_entries": 121, "num_filter_entries": 121, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1702712743, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "efa1eb90-5256-4a90-8760-5f545d6f2ce7", "db_session_id": "9A6KOWYETGATBZGNXD8K", "orig_file_number": 39, "seqno_to_time_mapping": "N/A"}} 2023-12-16T07:45:44.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.735+0000 7f4f68746c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702712743739900, "job": 1, "event": "recovery_finished"} 2023-12-16T07:45:44.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.735+0000 7f4f68746c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 41 2023-12-16T07:45:44.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.735+0000 7f4f68746c80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-12-16T07:45:44.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.735+0000 7f4f68746c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000034.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-12-16T07:45:44.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.735+0000 7f4f68746c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56293dbf2000 2023-12-16T07:45:44.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.735+0000 7f4f68746c80 4 rocksdb: DB pointer 0x56293dbe0000 2023-12-16T07:45:44.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.739+0000 7f4f56f00700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-12-16T07:45:44.087 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.739+0000 7f4f56f00700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2023-12-16T07:45:44.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: ** DB Stats ** 2023-12-16T07:45:44.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:45:44.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-16T07:45:44.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-16T07:45:44.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T07:45:44.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-16T07:45:44.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-16T07:45:44.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T07:45:44.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: ** Compaction Stats [default] ** 2023-12-16T07:45:44.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-12-16T07:45:44.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-12-16T07:45:44.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: L0 1/0 439.48 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2023-12-16T07:45:44.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: L6 1/0 9.93 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2023-12-16T07:45:44.088 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Sum 2/0 10.36 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2023-12-16T07:45:44.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2023-12-16T07:45:44.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: ** Compaction Stats [default] ** 2023-12-16T07:45:44.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-12-16T07:45:44.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:45:44.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 128.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2023-12-16T07:45:44.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-12-16T07:45:44.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:45:44.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-16T07:45:44.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T07:45:44.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T07:45:44.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T07:45:44.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: AddFile(Keys): cumulative 0, interval 0 2023-12-16T07:45:44.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Cumulative compaction: 0.00 GB write, 42.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:45:44.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Interval compaction: 0.00 GB write, 42.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:45:44.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T07:45:44.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Block cache BinnedLRUCache@0x56293db53090#7 capacity: 512.00 MB usage: 52.92 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.8e-05 secs_since: 0 2023-12-16T07:45:44.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: Block cache entry stats(count,size,portion): DataBlock(3,19.73 KB,0.00376403%) FilterBlock(2,12.41 KB,0.0023663%) IndexBlock(2,20.78 KB,0.00396371%) Misc(1,0.00 KB,0%) 2023-12-16T07:45:44.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: ** File Read Latency Histogram By Level [default] ** 2023-12-16T07:45:44.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.739+0000 7f4f68746c80 0 starting mon.a rank 0 at public addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] at bind addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:45:44.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.739+0000 7f4f68746c80 1 mon.a@-1(???) e3 preinit fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:45:44.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.739+0000 7f4f68746c80 0 mon.a@-1(???).mds e1 new map 2023-12-16T07:45:44.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.739+0000 7f4f68746c80 0 mon.a@-1(???).mds e1 print_map 2023-12-16T07:45:44.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: e1 2023-12-16T07:45:44.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: enable_multiple, ever_enabled_multiple: 1,1 2023-12-16T07:45:44.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T07:45:44.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: legacy client fscid: -1 2023-12-16T07:45:44.090 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: 2023-12-16T07:45:44.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: No filesystems configured 2023-12-16T07:45:44.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.739+0000 7f4f68746c80 0 mon.a@-1(???).osd e102 crush map has features 3314933000854323200, adjusting msgr requires 2023-12-16T07:45:44.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.739+0000 7f4f68746c80 0 mon.a@-1(???).osd e102 crush map has features 432629239337189376, adjusting msgr requires 2023-12-16T07:45:44.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.739+0000 7f4f68746c80 0 mon.a@-1(???).osd e102 crush map has features 432629239337189376, adjusting msgr requires 2023-12-16T07:45:44.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.739+0000 7f4f68746c80 0 mon.a@-1(???).osd e102 crush map has features 432629239337189376, adjusting msgr requires 2023-12-16T07:45:44.091 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:43 smithi007 bash[64479]: debug 2023-12-16T07:45:43.739+0000 7f4f68746c80 1 mon.a@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2023-12-16T07:45:44.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[61260]: ignoring --setuser ceph since I am not root 2023-12-16T07:45:44.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[61260]: ignoring --setgroup ceph since I am not root 2023-12-16T07:45:44.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[61260]: debug 2023-12-16T07:45:44.483+0000 7f2d20bc0700 1 -- 172.21.15.7:0/3540129830 <== mon.1 v2:172.21.15.7:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e79c6c7520 con 0x55e79d442c00 2023-12-16T07:45:44.818 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[61260]: debug 2023-12-16T07:45:44.643+0000 7f2d2a949200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T07:45:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[30436]: audit 2023-12-16T07:45:41.293123+0000 mon.b (mon.2) 94 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:45:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[30436]: audit 2023-12-16T07:45:41.294305+0000 mon.b (mon.2) 95 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:45:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[30436]: audit 2023-12-16T07:45:41.294734+0000 mon.b (mon.2) 96 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:45:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[30436]: cluster 2023-12-16T07:45:42.725001+0000 mgr.y (mgr.15429) 71 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[30436]: audit 2023-12-16T07:45:42.858827+0000 mon.b (mon.2) 97 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:45:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[30436]: cluster 2023-12-16T07:45:44.364803+0000 mon.a (mon.0) 1 : cluster [INF] mon.a calling monitor election 2023-12-16T07:45:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[30436]: cluster 2023-12-16T07:45:44.386466+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-12-16T07:45:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[30436]: cluster 2023-12-16T07:45:44.398501+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T07:45:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[30436]: cluster 2023-12-16T07:45:44.399832+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2023-12-16T07:45:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[30436]: cluster 2023-12-16T07:45:44.399911+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2023-12-16T07:45:44.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[30436]: cluster 2023-12-16T07:45:44.401400+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e33: y(active, since 62s), standbys: x 2023-12-16T07:45:44.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[30436]: audit 2023-12-16T07:45:44.415318+0000 mon.a (mon.0) 7 : audit [INF] from='mgr.15429 ' entity='' 2023-12-16T07:45:44.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[30436]: cluster 2023-12-16T07:45:44.416455+0000 mon.a (mon.0) 8 : cluster [DBG] mgrmap e34: y(active, since 62s), standbys: x 2023-12-16T07:45:44.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[30436]: cluster 2023-12-16T07:45:44.416522+0000 mon.a (mon.0) 9 : cluster [INF] overall HEALTH_OK 2023-12-16T07:45:44.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[64479]: audit 2023-12-16T07:45:41.293123+0000 mon.b (mon.2) 94 : audit 0 from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:45:44.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[64479]: audit 2023-12-16T07:45:41.294305+0000 mon.b (mon.2) 95 : audit 0 from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:45:44.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[64479]: audit 2023-12-16T07:45:41.294734+0000 mon.b (mon.2) 96 : audit 0 from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:45:44.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[64479]: cluster 2023-12-16T07:45:42.725001+0000 mgr.y (mgr.15429) 71 : cluster 0 pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:44.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[64479]: audit 2023-12-16T07:45:42.858827+0000 mon.b (mon.2) 97 : audit 0 from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:45:44.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[64479]: cluster 2023-12-16T07:45:44.364803+0000 mon.a (mon.0) 1 : cluster 1 mon.a calling monitor election 2023-12-16T07:45:44.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[64479]: cluster 2023-12-16T07:45:44.386466+0000 mon.a (mon.0) 2 : cluster 1 mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-12-16T07:45:44.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[64479]: cluster 2023-12-16T07:45:44.398501+0000 mon.a (mon.0) 3 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T07:45:44.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[64479]: cluster 2023-12-16T07:45:44.399832+0000 mon.a (mon.0) 4 : cluster 0 fsmap 2023-12-16T07:45:44.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[64479]: cluster 2023-12-16T07:45:44.399911+0000 mon.a (mon.0) 5 : cluster 0 osdmap e102: 8 total, 8 up, 8 in 2023-12-16T07:45:44.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[64479]: cluster 2023-12-16T07:45:44.401400+0000 mon.a (mon.0) 6 : cluster 0 mgrmap e33: y(active, since 62s), standbys: x 2023-12-16T07:45:44.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[64479]: audit 2023-12-16T07:45:44.415318+0000 mon.a (mon.0) 7 : audit 1 from='mgr.15429 ' entity='' 2023-12-16T07:45:44.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[64479]: cluster 2023-12-16T07:45:44.416455+0000 mon.a (mon.0) 8 : cluster 0 mgrmap e34: y(active, since 62s), standbys: x 2023-12-16T07:45:44.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:44 smithi007 bash[64479]: cluster 2023-12-16T07:45:44.416522+0000 mon.a (mon.0) 9 : cluster 1 overall HEALTH_OK 2023-12-16T07:45:44.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[49542]: ignoring --setuser ceph since I am not root 2023-12-16T07:45:44.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[49542]: ignoring --setgroup ceph since I am not root 2023-12-16T07:45:44.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[49542]: debug 2023-12-16T07:45:44.477+0000 7f0eaf987700 1 -- 172.21.15.169:0/500395755 <== mon.1 v2:172.21.15.7:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55949a851520 con 0x55949b5cc800 2023-12-16T07:45:44.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[49542]: debug 2023-12-16T07:45:44.637+0000 7f0eb9710200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-16T07:45:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[25083]: audit 2023-12-16T07:45:41.293123+0000 mon.b (mon.2) 94 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:45:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[25083]: audit 2023-12-16T07:45:41.294305+0000 mon.b (mon.2) 95 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:45:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[25083]: audit 2023-12-16T07:45:41.294734+0000 mon.b (mon.2) 96 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:45:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[25083]: cluster 2023-12-16T07:45:42.725001+0000 mgr.y (mgr.15429) 71 : cluster [DBG] pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:45:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[25083]: audit 2023-12-16T07:45:42.858827+0000 mon.b (mon.2) 97 : audit [DBG] from='mgr.15429 172.21.15.7:0/3209269934' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:45:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[25083]: cluster 2023-12-16T07:45:44.364803+0000 mon.a (mon.0) 1 : cluster [INF] mon.a calling monitor election 2023-12-16T07:45:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[25083]: cluster 2023-12-16T07:45:44.386466+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-12-16T07:45:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[25083]: cluster 2023-12-16T07:45:44.398501+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T07:45:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[25083]: cluster 2023-12-16T07:45:44.399832+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2023-12-16T07:45:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[25083]: cluster 2023-12-16T07:45:44.399911+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2023-12-16T07:45:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[25083]: cluster 2023-12-16T07:45:44.401400+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e33: y(active, since 62s), standbys: x 2023-12-16T07:45:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[25083]: audit 2023-12-16T07:45:44.415318+0000 mon.a (mon.0) 7 : audit [INF] from='mgr.15429 ' entity='' 2023-12-16T07:45:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[25083]: cluster 2023-12-16T07:45:44.416455+0000 mon.a (mon.0) 8 : cluster [DBG] mgrmap e34: y(active, since 62s), standbys: x 2023-12-16T07:45:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:44 smithi169 bash[25083]: cluster 2023-12-16T07:45:44.416522+0000 mon.a (mon.0) 9 : cluster [INF] overall HEALTH_OK 2023-12-16T07:45:45.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:45 smithi169 bash[49542]: debug 2023-12-16T07:45:45.533+0000 7f0eb9710200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-16T07:45:45.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:45 smithi169 bash[49542]: debug 2023-12-16T07:45:45.633+0000 7f0eb9710200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T07:45:45.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:45 smithi169 bash[49542]: debug 2023-12-16T07:45:45.733+0000 7f0eb9710200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T07:45:46.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:45 smithi007 bash[61260]: debug 2023-12-16T07:45:45.611+0000 7f2d2a949200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-16T07:45:46.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:45 smithi007 bash[61260]: debug 2023-12-16T07:45:45.723+0000 7f2d2a949200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-16T07:45:46.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:45 smithi007 bash[61260]: debug 2023-12-16T07:45:45.835+0000 7f2d2a949200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-16T07:45:46.362 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:46 smithi169 bash[49542]: debug 2023-12-16T07:45:46.001+0000 7f0eb9710200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:45:46.419 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:46 smithi007 bash[61260]: debug 2023-12-16T07:45:46.131+0000 7f2d2a949200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:45:46.679 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:46 smithi007 bash[61260]: debug 2023-12-16T07:45:46.511+0000 7f2d2a949200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T07:45:46.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:46 smithi007 bash[30436]: cluster 2023-12-16T07:45:45.422929+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e35: y(active, since 63s), standbys: x 2023-12-16T07:45:46.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:46 smithi007 bash[64479]: cluster 2023-12-16T07:45:45.422929+0000 mon.a (mon.0) 10 : cluster 0 mgrmap e35: y(active, since 63s), standbys: x 2023-12-16T07:45:46.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:46 smithi169 bash[25083]: cluster 2023-12-16T07:45:45.422929+0000 mon.a (mon.0) 10 : cluster [DBG] mgrmap e35: y(active, since 63s), standbys: x 2023-12-16T07:45:46.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:46 smithi169 bash[49542]: debug 2023-12-16T07:45:46.361+0000 7f0eb9710200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-16T07:45:46.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:46 smithi169 bash[49542]: debug 2023-12-16T07:45:46.521+0000 7f0eb9710200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T07:45:47.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:46 smithi007 bash[61260]: debug 2023-12-16T07:45:46.671+0000 7f2d2a949200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-16T07:45:47.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:46 smithi007 bash[61260]: debug 2023-12-16T07:45:46.891+0000 7f2d2a949200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T07:45:47.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:46 smithi169 bash[49542]: debug 2023-12-16T07:45:46.729+0000 7f0eb9710200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-16T07:45:48.184 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:47 smithi169 bash[49542]: debug 2023-12-16T07:45:47.725+0000 7f0eb9710200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T07:45:48.184 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:47 smithi169 bash[49542]: debug 2023-12-16T07:45:47.833+0000 7f0eb9710200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T07:45:48.185 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:47 smithi169 bash[49542]: debug 2023-12-16T07:45:47.949+0000 7f0eb9710200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T07:45:48.318 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:47 smithi007 bash[61260]: debug 2023-12-16T07:45:47.923+0000 7f2d2a949200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-16T07:45:48.319 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:48 smithi007 bash[61260]: debug 2023-12-16T07:45:48.039+0000 7f2d2a949200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-16T07:45:48.319 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:48 smithi007 bash[61260]: debug 2023-12-16T07:45:48.159+0000 7f2d2a949200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-16T07:45:48.683 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:45:48 smithi169 bash[49087]: ts=2023-12-16T07:45:48.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:45:48.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:48 smithi169 bash[49542]: debug 2023-12-16T07:45:48.421+0000 7f0eb9710200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T07:45:48.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:48 smithi169 bash[49542]: debug 2023-12-16T07:45:48.649+0000 7f0eb9710200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T07:45:49.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:48 smithi007 bash[61260]: debug 2023-12-16T07:45:48.651+0000 7f2d2a949200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-16T07:45:49.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:48 smithi007 bash[61260]: debug 2023-12-16T07:45:48.887+0000 7f2d2a949200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-16T07:45:49.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:49 smithi169 bash[49542]: debug 2023-12-16T07:45:49.105+0000 7f0eb9710200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T07:45:49.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:49 smithi007 bash[61260]: debug 2023-12-16T07:45:49.355+0000 7f2d2a949200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-16T07:45:49.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:49 smithi169 bash[49542]: debug 2023-12-16T07:45:49.601+0000 7f0eb9710200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:45:50.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:49 smithi007 bash[61260]: debug 2023-12-16T07:45:49.863+0000 7f2d2a949200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-16T07:45:50.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:50 smithi169 bash[49542]: debug 2023-12-16T07:45:50.141+0000 7f0eb9710200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T07:45:50.433 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:50 smithi169 bash[49542]: debug 2023-12-16T07:45:50.249+0000 7f0eb9710200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T07:45:50.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:50 smithi007 bash[61260]: debug 2023-12-16T07:45:50.435+0000 7f2d2a949200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-16T07:45:50.817 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:50 smithi007 bash[61260]: debug 2023-12-16T07:45:50.547+0000 7f2d2a949200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-16T07:45:50.860 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:50 smithi169 bash[49542]: debug 2023-12-16T07:45:50.601+0000 7f0eb9710200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T07:45:50.860 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:50 smithi169 bash[49542]: debug 2023-12-16T07:45:50.713+0000 7f0eb9710200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T07:45:51.180 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:50 smithi007 bash[61260]: debug 2023-12-16T07:45:50.903+0000 7f2d2a949200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-16T07:45:51.181 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:51 smithi007 bash[61260]: debug 2023-12-16T07:45:51.023+0000 7f2d2a949200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-16T07:45:51.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:50 smithi169 bash[49542]: debug 2023-12-16T07:45:50.857+0000 7f0eb9710200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T07:45:51.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:50 smithi169 bash[49542]: debug 2023-12-16T07:45:50.973+0000 7f0eb9710200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T07:45:51.183 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:51 smithi169 bash[49542]: debug 2023-12-16T07:45:51.097+0000 7f0eb9710200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T07:45:51.433 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:51 smithi007 bash[61260]: debug 2023-12-16T07:45:51.175+0000 7f2d2a949200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-16T07:45:51.433 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:51 smithi007 bash[61260]: debug 2023-12-16T07:45:51.295+0000 7f2d2a949200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-16T07:45:51.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:51 smithi169 bash[49542]: debug 2023-12-16T07:45:51.285+0000 7f0eb9710200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T07:45:51.683 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:51 smithi169 bash[49542]: debug 2023-12-16T07:45:51.405+0000 7f0eb9710200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T07:45:51.759 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:51 smithi007 bash[61260]: debug 2023-12-16T07:45:51.427+0000 7f2d2a949200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-16T07:45:51.759 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:51 smithi007 bash[61260]: debug 2023-12-16T07:45:51.623+0000 7f2d2a949200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-16T07:45:52.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:51 smithi007 bash[61260]: debug 2023-12-16T07:45:51.751+0000 7f2d2a949200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-16T07:45:52.933 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:52 smithi169 bash[49542]: debug 2023-12-16T07:45:52.645+0000 7f0eb9710200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T07:45:52.934 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:52 smithi169 bash[49542]: [16/Dec/2023:07:45:52] ENGINE Bus STARTING 2023-12-16T07:45:52.934 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:52 smithi169 bash[49542]: CherryPy Checker: 2023-12-16T07:45:52.934 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:52 smithi169 bash[49542]: The Application mounted at '' has an empty config. 2023-12-16T07:45:52.934 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:52 smithi169 bash[49542]: [16/Dec/2023:07:45:52] ENGINE Serving on http://:::9283 2023-12-16T07:45:52.934 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:45:52 smithi169 bash[49542]: [16/Dec/2023:07:45:52] ENGINE Bus STARTED 2023-12-16T07:45:52.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:52 smithi169 bash[25083]: cluster 2023-12-16T07:45:52.666436+0000 mon.a (mon.0) 11 : cluster [DBG] Standby manager daemon x restarted 2023-12-16T07:45:52.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:52 smithi169 bash[25083]: cluster 2023-12-16T07:45:52.666577+0000 mon.a (mon.0) 12 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:45:52.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:52 smithi169 bash[25083]: audit 2023-12-16T07:45:52.670925+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.? 172.21.15.169:0/2513825601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:45:52.936 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:52 smithi169 bash[25083]: audit 2023-12-16T07:45:52.673109+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.? 172.21.15.169:0/2513825601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:45:52.936 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:52 smithi169 bash[25083]: audit 2023-12-16T07:45:52.680196+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.? 172.21.15.169:0/2513825601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:45:52.936 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:52 smithi169 bash[25083]: audit 2023-12-16T07:45:52.681499+0000 mon.a (mon.0) 16 : audit [DBG] from='mgr.? 172.21.15.169:0/2513825601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:45:53.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:52 smithi007 bash[30436]: cluster 2023-12-16T07:45:52.666436+0000 mon.a (mon.0) 11 : cluster [DBG] Standby manager daemon x restarted 2023-12-16T07:45:53.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:52 smithi007 bash[30436]: cluster 2023-12-16T07:45:52.666577+0000 mon.a (mon.0) 12 : cluster [DBG] Standby manager daemon x started 2023-12-16T07:45:53.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:52 smithi007 bash[30436]: audit 2023-12-16T07:45:52.670925+0000 mon.a (mon.0) 13 : audit [DBG] from='mgr.? 172.21.15.169:0/2513825601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:45:53.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:52 smithi007 bash[30436]: audit 2023-12-16T07:45:52.673109+0000 mon.a (mon.0) 14 : audit [DBG] from='mgr.? 172.21.15.169:0/2513825601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:45:53.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:52 smithi007 bash[30436]: audit 2023-12-16T07:45:52.680196+0000 mon.a (mon.0) 15 : audit [DBG] from='mgr.? 172.21.15.169:0/2513825601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:45:53.032 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:52 smithi007 bash[30436]: audit 2023-12-16T07:45:52.681499+0000 mon.a (mon.0) 16 : audit [DBG] from='mgr.? 172.21.15.169:0/2513825601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:45:53.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:52 smithi007 bash[64479]: cluster 2023-12-16T07:45:52.666436+0000 mon.a (mon.0) 11 : cluster 0 Standby manager daemon x restarted 2023-12-16T07:45:53.032 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:52 smithi007 bash[64479]: cluster 2023-12-16T07:45:52.666577+0000 mon.a (mon.0) 12 : cluster 0 Standby manager daemon x started 2023-12-16T07:45:53.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:52 smithi007 bash[64479]: audit 2023-12-16T07:45:52.670925+0000 mon.a (mon.0) 13 : audit 0 from='mgr.? 172.21.15.169:0/2513825601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-12-16T07:45:53.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:52 smithi007 bash[64479]: audit 2023-12-16T07:45:52.673109+0000 mon.a (mon.0) 14 : audit 0 from='mgr.? 172.21.15.169:0/2513825601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-12-16T07:45:53.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:52 smithi007 bash[64479]: audit 2023-12-16T07:45:52.680196+0000 mon.a (mon.0) 15 : audit 0 from='mgr.? 172.21.15.169:0/2513825601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-12-16T07:45:53.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:52 smithi007 bash[64479]: audit 2023-12-16T07:45:52.681499+0000 mon.a (mon.0) 16 : audit 0 from='mgr.? 172.21.15.169:0/2513825601' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-12-16T07:45:53.286 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: debug 2023-12-16T07:45:53.051+0000 7fe9d816d700 -1 mon.c@1(peon).osd e103 update_from_paxos full map CRC mismatch, resetting to canonical 2023-12-16T07:45:53.286 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[61260]: debug 2023-12-16T07:45:53.023+0000 7f2d2a949200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-16T07:45:53.286 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[61260]: [16/Dec/2023:07:45:53] ENGINE Bus STARTING 2023-12-16T07:45:53.340 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: debug 2023-12-16T07:45:53.057+0000 7f4b3c9e5700 -1 mon.b@2(peon).osd e103 update_from_paxos full map CRC mismatch, resetting to canonical 2023-12-16T07:45:53.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[61260]: CherryPy Checker: 2023-12-16T07:45:53.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[61260]: The Application mounted at '' has an empty config. 2023-12-16T07:45:53.569 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[61260]: [16/Dec/2023:07:45:53] ENGINE Serving on http://:::9283 2023-12-16T07:45:53.569 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[61260]: [16/Dec/2023:07:45:53] ENGINE Bus STARTED 2023-12-16T07:45:54.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: cluster 2023-12-16T07:45:52.728962+0000 mon.a (mon.0) 17 : cluster 0 mgrmap e36: y(active, since 70s), standbys: x 2023-12-16T07:45:54.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: cluster 2023-12-16T07:45:53.050113+0000 mon.a (mon.0) 18 : cluster 1 Active manager daemon y restarted 2023-12-16T07:45:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: cluster 2023-12-16T07:45:53.050882+0000 mon.a (mon.0) 19 : cluster 1 Activating manager daemon y 2023-12-16T07:45:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: cluster 2023-12-16T07:45:53.062110+0000 mon.a (mon.0) 20 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2023-12-16T07:45:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: cluster 2023-12-16T07:45:53.062848+0000 mon.a (mon.0) 21 : cluster 0 mgrmap e37: y(active, starting, since 0.0121392s), standbys: x 2023-12-16T07:45:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: cluster 2023-12-16T07:45:53.065492+0000 osd.0 (osd.0) 3 : cluster 3 failed to encode map e103 with expected crc 2023-12-16T07:45:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: cluster 2023-12-16T07:45:53.068596+0000 osd.4 (osd.4) 3 : cluster 3 failed to encode map e103 with expected crc 2023-12-16T07:45:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.076674+0000 mon.a (mon.0) 22 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:45:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.077045+0000 mon.a (mon.0) 23 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:45:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.077306+0000 mon.a (mon.0) 24 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:45:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.079017+0000 mon.a (mon.0) 25 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:45:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.079444+0000 mon.a (mon.0) 26 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:45:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.079715+0000 mon.a (mon.0) 27 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:45:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.080011+0000 mon.a (mon.0) 28 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:45:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.080290+0000 mon.a (mon.0) 29 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:45:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.080603+0000 mon.a (mon.0) 30 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:45:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.080999+0000 mon.a (mon.0) 31 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:45:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.081365+0000 mon.a (mon.0) 32 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:45:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.081613+0000 mon.a (mon.0) 33 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:45:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.081918+0000 mon.a (mon.0) 34 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:45:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.082217+0000 mon.a (mon.0) 35 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:45:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.082424+0000 mon.a (mon.0) 36 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:45:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.082978+0000 mon.a (mon.0) 37 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:45:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: cluster 2023-12-16T07:45:53.091635+0000 mon.a (mon.0) 38 : cluster 1 Manager daemon y is now available 2023-12-16T07:45:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.139099+0000 mon.a (mon.0) 39 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:45:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: cluster 2023-12-16T07:45:53.177694+0000 osd.1 (osd.1) 3 : cluster 3 failed to encode map e103 with expected crc 2023-12-16T07:45:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: cluster 2023-12-16T07:45:53.177852+0000 osd.3 (osd.3) 3 : cluster 3 failed to encode map e103 with expected crc 2023-12-16T07:45:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.196664+0000 mon.a (mon.0) 40 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:45:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.211635+0000 mon.a (mon.0) 41 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:45:54.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: audit 2023-12-16T07:45:53.262064+0000 mon.a (mon.0) 42 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:45:54.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[64479]: cluster 2023-12-16T07:45:53.457628+0000 osd.2 (osd.2) 3 : cluster 3 failed to encode map e103 with expected crc 2023-12-16T07:45:54.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: cluster 2023-12-16T07:45:52.728962+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e36: y(active, since 70s), standbys: x 2023-12-16T07:45:54.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: cluster 2023-12-16T07:45:53.050113+0000 mon.a (mon.0) 18 : cluster [INF] Active manager daemon y restarted 2023-12-16T07:45:54.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: cluster 2023-12-16T07:45:53.050882+0000 mon.a (mon.0) 19 : cluster [INF] Activating manager daemon y 2023-12-16T07:45:54.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: cluster 2023-12-16T07:45:53.062110+0000 mon.a (mon.0) 20 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2023-12-16T07:45:54.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: cluster 2023-12-16T07:45:53.062848+0000 mon.a (mon.0) 21 : cluster [DBG] mgrmap e37: y(active, starting, since 0.0121392s), standbys: x 2023-12-16T07:45:54.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: cluster 2023-12-16T07:45:53.065492+0000 osd.0 (osd.0) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:54.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: cluster 2023-12-16T07:45:53.068596+0000 osd.4 (osd.4) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:54.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.076674+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:45:54.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.077045+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:45:54.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.077306+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:45:54.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.079017+0000 mon.a (mon.0) 25 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:45:54.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.079444+0000 mon.a (mon.0) 26 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:45:54.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.079715+0000 mon.a (mon.0) 27 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:45:54.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.080011+0000 mon.a (mon.0) 28 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:45:54.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.080290+0000 mon.a (mon.0) 29 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:45:54.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.080603+0000 mon.a (mon.0) 30 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:45:54.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.080999+0000 mon.a (mon.0) 31 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:45:54.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.081365+0000 mon.a (mon.0) 32 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:45:54.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.081613+0000 mon.a (mon.0) 33 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:45:54.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.081918+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:45:54.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.082217+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:45:54.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.082424+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:45:54.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.082978+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:45:54.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: cluster 2023-12-16T07:45:53.091635+0000 mon.a (mon.0) 38 : cluster [INF] Manager daemon y is now available 2023-12-16T07:45:54.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.139099+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:45:54.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: cluster 2023-12-16T07:45:53.177694+0000 osd.1 (osd.1) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:54.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: cluster 2023-12-16T07:45:53.177852+0000 osd.3 (osd.3) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:54.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.196664+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:45:54.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.211635+0000 mon.a (mon.0) 41 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:45:54.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: audit 2023-12-16T07:45:53.262064+0000 mon.a (mon.0) 42 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:45:54.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:53 smithi007 bash[30436]: cluster 2023-12-16T07:45:53.457628+0000 osd.2 (osd.2) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:54.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: cluster 2023-12-16T07:45:52.728962+0000 mon.a (mon.0) 17 : cluster [DBG] mgrmap e36: y(active, since 70s), standbys: x 2023-12-16T07:45:54.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: cluster 2023-12-16T07:45:53.050113+0000 mon.a (mon.0) 18 : cluster [INF] Active manager daemon y restarted 2023-12-16T07:45:54.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: cluster 2023-12-16T07:45:53.050882+0000 mon.a (mon.0) 19 : cluster [INF] Activating manager daemon y 2023-12-16T07:45:54.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: cluster 2023-12-16T07:45:53.062110+0000 mon.a (mon.0) 20 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2023-12-16T07:45:54.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: cluster 2023-12-16T07:45:53.062848+0000 mon.a (mon.0) 21 : cluster [DBG] mgrmap e37: y(active, starting, since 0.0121392s), standbys: x 2023-12-16T07:45:54.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: cluster 2023-12-16T07:45:53.065492+0000 osd.0 (osd.0) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:54.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: cluster 2023-12-16T07:45:53.068596+0000 osd.4 (osd.4) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:54.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.076674+0000 mon.a (mon.0) 22 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-12-16T07:45:54.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.077045+0000 mon.a (mon.0) 23 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-12-16T07:45:54.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.077306+0000 mon.a (mon.0) 24 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-12-16T07:45:54.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.079017+0000 mon.a (mon.0) 25 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-12-16T07:45:54.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.079444+0000 mon.a (mon.0) 26 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-12-16T07:45:54.186 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.079715+0000 mon.a (mon.0) 27 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:45:54.186 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.080011+0000 mon.a (mon.0) 28 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:45:54.186 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.080290+0000 mon.a (mon.0) 29 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:45:54.186 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.080603+0000 mon.a (mon.0) 30 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:45:54.186 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.080999+0000 mon.a (mon.0) 31 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:45:54.186 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.081365+0000 mon.a (mon.0) 32 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:45:54.186 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.081613+0000 mon.a (mon.0) 33 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:45:54.186 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.081918+0000 mon.a (mon.0) 34 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:45:54.186 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.082217+0000 mon.a (mon.0) 35 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-16T07:45:54.186 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.082424+0000 mon.a (mon.0) 36 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-16T07:45:54.186 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.082978+0000 mon.a (mon.0) 37 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-16T07:45:54.186 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: cluster 2023-12-16T07:45:53.091635+0000 mon.a (mon.0) 38 : cluster [INF] Manager daemon y is now available 2023-12-16T07:45:54.186 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.139099+0000 mon.a (mon.0) 39 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:45:54.187 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: cluster 2023-12-16T07:45:53.177694+0000 osd.1 (osd.1) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:54.187 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: cluster 2023-12-16T07:45:53.177852+0000 osd.3 (osd.3) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:54.187 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.196664+0000 mon.a (mon.0) 40 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:45:54.187 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.211635+0000 mon.a (mon.0) 41 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-12-16T07:45:54.187 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: audit 2023-12-16T07:45:53.262064+0000 mon.a (mon.0) 42 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-12-16T07:45:54.187 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:53 smithi169 bash[25083]: cluster 2023-12-16T07:45:53.457628+0000 osd.2 (osd.2) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:54.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:45:54 smithi007 bash[61260]: debug 2023-12-16T07:45:54.067+0000 7f2cea7a7700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-16T07:45:55.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:55 smithi169 bash[25083]: cluster 2023-12-16T07:45:53.180195+0000 osd.5 (osd.5) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:55.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:55 smithi169 bash[25083]: cluster 2023-12-16T07:45:53.180476+0000 osd.6 (osd.6) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:55.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:55 smithi169 bash[25083]: cluster 2023-12-16T07:45:53.220978+0000 osd.7 (osd.7) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:55.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:55 smithi169 bash[25083]: cluster 2023-12-16T07:45:54.068925+0000 mon.a (mon.0) 43 : cluster [DBG] mgrmap e38: y(active, since 1.01822s), standbys: x 2023-12-16T07:45:55.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:55 smithi169 bash[25083]: audit 2023-12-16T07:45:54.082339+0000 mgr.y (mgr.25477) 1 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:55.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:55 smithi169 bash[25083]: cluster 2023-12-16T07:45:54.085194+0000 mgr.y (mgr.25477) 2 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:45:55.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:55 smithi007 bash[30436]: cluster 2023-12-16T07:45:53.180195+0000 osd.5 (osd.5) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:55.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:55 smithi007 bash[30436]: cluster 2023-12-16T07:45:53.180476+0000 osd.6 (osd.6) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:55.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:55 smithi007 bash[30436]: cluster 2023-12-16T07:45:53.220978+0000 osd.7 (osd.7) 3 : cluster [WRN] failed to encode map e103 with expected crc 2023-12-16T07:45:55.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:55 smithi007 bash[30436]: cluster 2023-12-16T07:45:54.068925+0000 mon.a (mon.0) 43 : cluster [DBG] mgrmap e38: y(active, since 1.01822s), standbys: x 2023-12-16T07:45:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:55 smithi007 bash[30436]: audit 2023-12-16T07:45:54.082339+0000 mgr.y (mgr.25477) 1 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:55 smithi007 bash[30436]: cluster 2023-12-16T07:45:54.085194+0000 mgr.y (mgr.25477) 2 : cluster [DBG] pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:45:55.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:55 smithi007 bash[64479]: cluster 2023-12-16T07:45:53.180195+0000 osd.5 (osd.5) 3 : cluster 3 failed to encode map e103 with expected crc 2023-12-16T07:45:55.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:55 smithi007 bash[64479]: cluster 2023-12-16T07:45:53.180476+0000 osd.6 (osd.6) 3 : cluster 3 failed to encode map e103 with expected crc 2023-12-16T07:45:55.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:55 smithi007 bash[64479]: cluster 2023-12-16T07:45:53.220978+0000 osd.7 (osd.7) 3 : cluster 3 failed to encode map e103 with expected crc 2023-12-16T07:45:55.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:55 smithi007 bash[64479]: cluster 2023-12-16T07:45:54.068925+0000 mon.a (mon.0) 43 : cluster 0 mgrmap e38: y(active, since 1.01822s), standbys: x 2023-12-16T07:45:55.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:55 smithi007 bash[64479]: audit 2023-12-16T07:45:54.082339+0000 mgr.y (mgr.25477) 1 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:45:55.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:55 smithi007 bash[64479]: cluster 2023-12-16T07:45:54.085194+0000 mgr.y (mgr.25477) 2 : cluster 0 pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:45:56.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:56 smithi169 bash[25083]: cephadm 2023-12-16T07:45:54.665154+0000 mgr.y (mgr.25477) 3 : cephadm [INF] [16/Dec/2023:07:45:54] ENGINE Bus STARTING 2023-12-16T07:45:56.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:56 smithi169 bash[25083]: cephadm 2023-12-16T07:45:54.767618+0000 mgr.y (mgr.25477) 4 : cephadm [INF] [16/Dec/2023:07:45:54] ENGINE Serving on http://172.21.15.7:8765 2023-12-16T07:45:56.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:56 smithi169 bash[25083]: cephadm 2023-12-16T07:45:54.901237+0000 mgr.y (mgr.25477) 5 : cephadm [INF] [16/Dec/2023:07:45:54] ENGINE Serving on https://172.21.15.7:7150 2023-12-16T07:45:56.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:56 smithi169 bash[25083]: cephadm 2023-12-16T07:45:54.901770+0000 mgr.y (mgr.25477) 6 : cephadm [INF] [16/Dec/2023:07:45:54] ENGINE Bus STARTED 2023-12-16T07:45:56.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:56 smithi169 bash[25083]: cluster 2023-12-16T07:45:55.080896+0000 mgr.y (mgr.25477) 7 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:45:56.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:56 smithi007 bash[64479]: cephadm 2023-12-16T07:45:54.665154+0000 mgr.y (mgr.25477) 3 : cephadm 1 [16/Dec/2023:07:45:54] ENGINE Bus STARTING 2023-12-16T07:45:56.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:56 smithi007 bash[64479]: cephadm 2023-12-16T07:45:54.767618+0000 mgr.y (mgr.25477) 4 : cephadm 1 [16/Dec/2023:07:45:54] ENGINE Serving on http://172.21.15.7:8765 2023-12-16T07:45:56.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:56 smithi007 bash[64479]: cephadm 2023-12-16T07:45:54.901237+0000 mgr.y (mgr.25477) 5 : cephadm 1 [16/Dec/2023:07:45:54] ENGINE Serving on https://172.21.15.7:7150 2023-12-16T07:45:56.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:56 smithi007 bash[64479]: cephadm 2023-12-16T07:45:54.901770+0000 mgr.y (mgr.25477) 6 : cephadm 1 [16/Dec/2023:07:45:54] ENGINE Bus STARTED 2023-12-16T07:45:56.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:56 smithi007 bash[64479]: cluster 2023-12-16T07:45:55.080896+0000 mgr.y (mgr.25477) 7 : cluster 0 pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:45:56.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:56 smithi007 bash[30436]: cephadm 2023-12-16T07:45:54.665154+0000 mgr.y (mgr.25477) 3 : cephadm [INF] [16/Dec/2023:07:45:54] ENGINE Bus STARTING 2023-12-16T07:45:56.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:56 smithi007 bash[30436]: cephadm 2023-12-16T07:45:54.767618+0000 mgr.y (mgr.25477) 4 : cephadm [INF] [16/Dec/2023:07:45:54] ENGINE Serving on http://172.21.15.7:8765 2023-12-16T07:45:56.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:56 smithi007 bash[30436]: cephadm 2023-12-16T07:45:54.901237+0000 mgr.y (mgr.25477) 5 : cephadm [INF] [16/Dec/2023:07:45:54] ENGINE Serving on https://172.21.15.7:7150 2023-12-16T07:45:56.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:56 smithi007 bash[30436]: cephadm 2023-12-16T07:45:54.901770+0000 mgr.y (mgr.25477) 6 : cephadm [INF] [16/Dec/2023:07:45:54] ENGINE Bus STARTED 2023-12-16T07:45:56.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:56 smithi007 bash[30436]: cluster 2023-12-16T07:45:55.080896+0000 mgr.y (mgr.25477) 7 : cluster [DBG] pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:45:57.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:57 smithi169 bash[25083]: cluster 2023-12-16T07:45:56.077860+0000 mon.a (mon.0) 44 : cluster [DBG] mgrmap e39: y(active, since 3s), standbys: x 2023-12-16T07:45:57.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:57 smithi007 bash[64479]: cluster 2023-12-16T07:45:56.077860+0000 mon.a (mon.0) 44 : cluster 0 mgrmap e39: y(active, since 3s), standbys: x 2023-12-16T07:45:57.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:57 smithi007 bash[30436]: cluster 2023-12-16T07:45:56.077860+0000 mon.a (mon.0) 44 : cluster [DBG] mgrmap e39: y(active, since 3s), standbys: x 2023-12-16T07:45:58.044 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:45:58.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:58 smithi169 bash[25083]: cluster 2023-12-16T07:45:57.081747+0000 mgr.y (mgr.25477) 8 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:45:58.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:58 smithi169 bash[25083]: cluster 2023-12-16T07:45:57.155872+0000 mon.a (mon.0) 45 : cluster [DBG] mgrmap e40: y(active, since 4s), standbys: x 2023-12-16T07:45:58.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:58 smithi007 bash[30436]: cluster 2023-12-16T07:45:57.081747+0000 mgr.y (mgr.25477) 8 : cluster [DBG] pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:45:58.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:58 smithi007 bash[30436]: cluster 2023-12-16T07:45:57.155872+0000 mon.a (mon.0) 45 : cluster [DBG] mgrmap e40: y(active, since 4s), standbys: x 2023-12-16T07:45:58.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:58 smithi007 bash[64479]: cluster 2023-12-16T07:45:57.081747+0000 mgr.y (mgr.25477) 8 : cluster 0 pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:45:58.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:58 smithi007 bash[64479]: cluster 2023-12-16T07:45:57.155872+0000 mon.a (mon.0) 45 : cluster 0 mgrmap e40: y(active, since 4s), standbys: x 2023-12-16T07:45:58.846 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:45:58.846 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (2m) 34s ago 9m 16.1M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:45:58.846 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (101s) 34s ago 9m 43.2M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:45:58.846 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (52s) 34s ago 7m 44.0M - 3.5 e1d6a67b021e 8d8009d0c83f 2023-12-16T07:45:58.846 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283,8765 running (40s) 34s ago 15m 94.7M - 19.0.0-191-g6d933094 a29c258b5d6f 5295ba35c2cd 2023-12-16T07:45:58.846 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (95s) 34s ago 17m 483M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:45:58.846 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (17m) 34s ago 17m 61.0M 2048M 17.2.0 e1d6a67b021e c0468ecbd44e 2023-12-16T07:45:58.846 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (15m) 34s ago 15m 60.5M 2048M 17.2.0 e1d6a67b021e b47a960c6102 2023-12-16T07:45:58.846 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (15m) 34s ago 15m 55.5M 2048M 17.2.0 e1d6a67b021e c659b986f201 2023-12-16T07:45:58.847 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (2m) 34s ago 9m 10.7M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:45:58.847 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (2m) 34s ago 9m 11.2M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:45:58.847 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (15m) 34s ago 15m 57.8M 2782M 17.2.0 e1d6a67b021e dbc4d9538238 2023-12-16T07:45:58.847 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (14m) 34s ago 14m 58.4M 2782M 17.2.0 e1d6a67b021e 14637e0f389c 2023-12-16T07:45:58.847 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (13m) 34s ago 13m 56.8M 2782M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:45:58.847 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (13m) 34s ago 13m 56.5M 2782M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:45:58.847 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (12m) 34s ago 12m 57.9M 3037M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:45:58.847 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (11m) 34s ago 12m 54.5M 3037M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:45:58.847 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (11m) 34s ago 11m 54.9M 3037M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:45:58.847 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (10m) 34s ago 10m 55.7M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:45:58.847 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (47s) 34s ago 9m 47.9M - 2.33.4 514e6a882f6e 2652ce1b8540 2023-12-16T07:45:58.847 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (7m) 34s ago 7m 86.5M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:45:58.847 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (7m) 34s ago 7m 86.4M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:45:58.847 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (7m) 34s ago 7m 86.1M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:45:58.848 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (7m) 34s ago 7m 86.3M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:45:59.326 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:45:59.326 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:45:59.326 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2023-12-16T07:45:59.326 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 1 2023-12-16T07:45:59.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:45:59.326 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:45:59.326 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:45:59.326 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:45:59.326 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:45:59.327 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2023-12-16T07:45:59.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:45:59.327 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:45:59.327 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:45:59.327 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:45:59.327 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:45:59.327 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2023-12-16T07:45:59.327 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 3 2023-12-16T07:45:59.327 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:45:59.327 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:45:59.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:59 smithi169 bash[25083]: audit 2023-12-16T07:45:58.026693+0000 mgr.y (mgr.25477) 9 : audit [DBG] from='client.25507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:59.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:45:59 smithi169 bash[25083]: audit 2023-12-16T07:45:58.424232+0000 mgr.y (mgr.25477) 10 : audit [DBG] from='client.34137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:59.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:59 smithi007 bash[30436]: audit 2023-12-16T07:45:58.026693+0000 mgr.y (mgr.25477) 9 : audit [DBG] from='client.25507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:59.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:45:59 smithi007 bash[30436]: audit 2023-12-16T07:45:58.424232+0000 mgr.y (mgr.25477) 10 : audit [DBG] from='client.34137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:59.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:59 smithi007 bash[64479]: audit 2023-12-16T07:45:58.026693+0000 mgr.y (mgr.25477) 9 : audit 0 from='client.25507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:59.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:45:59 smithi007 bash[64479]: audit 2023-12-16T07:45:58.424232+0000 mgr.y (mgr.25477) 10 : audit 0 from='client.34137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:45:59.769 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:45:59.770 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:45:59.770 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:45:59.770 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:45:59.770 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2023-12-16T07:45:59.770 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2023-12-16T07:45:59.770 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:45:59.770 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "2/23 daemons upgraded", 2023-12-16T07:45:59.770 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "", 2023-12-16T07:45:59.770 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:45:59.770 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:46:00.263 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:46:00.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:00 smithi169 bash[25083]: audit 2023-12-16T07:45:58.827695+0000 mgr.y (mgr.25477) 11 : audit [DBG] from='client.25519 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:00.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:00 smithi169 bash[25083]: cluster 2023-12-16T07:45:59.082980+0000 mgr.y (mgr.25477) 12 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2023-12-16T07:46:00.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:00 smithi169 bash[25083]: audit 2023-12-16T07:45:59.323688+0000 mon.a (mon.0) 46 : audit [DBG] from='client.? 172.21.15.7:0/789007946' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:46:00.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:00 smithi169 bash[25083]: audit 2023-12-16T07:46:00.028070+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:00.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:00 smithi169 bash[25083]: audit 2023-12-16T07:46:00.036365+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:00.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:00 smithi169 bash[25083]: audit 2023-12-16T07:46:00.042432+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:00.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:00 smithi169 bash[25083]: audit 2023-12-16T07:46:00.050174+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:00.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[30436]: audit 2023-12-16T07:45:58.827695+0000 mgr.y (mgr.25477) 11 : audit [DBG] from='client.25519 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:00.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[30436]: cluster 2023-12-16T07:45:59.082980+0000 mgr.y (mgr.25477) 12 : cluster [DBG] pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2023-12-16T07:46:00.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[30436]: audit 2023-12-16T07:45:59.323688+0000 mon.a (mon.0) 46 : audit [DBG] from='client.? 172.21.15.7:0/789007946' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:46:00.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[30436]: audit 2023-12-16T07:46:00.028070+0000 mon.a (mon.0) 47 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:00.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[30436]: audit 2023-12-16T07:46:00.036365+0000 mon.a (mon.0) 48 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:00.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[30436]: audit 2023-12-16T07:46:00.042432+0000 mon.a (mon.0) 49 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:00.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[30436]: audit 2023-12-16T07:46:00.050174+0000 mon.a (mon.0) 50 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:00.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:46:00] "GET /metrics HTTP/1.1" 200 35356 "" "Prometheus/2.33.4" 2023-12-16T07:46:00.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[64479]: audit 2023-12-16T07:45:58.827695+0000 mgr.y (mgr.25477) 11 : audit 0 from='client.25519 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:00.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[64479]: cluster 2023-12-16T07:45:59.082980+0000 mgr.y (mgr.25477) 12 : cluster 0 pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2023-12-16T07:46:00.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[64479]: audit 2023-12-16T07:45:59.323688+0000 mon.a (mon.0) 46 : audit 0 from='client.? 172.21.15.7:0/789007946' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:46:00.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[64479]: audit 2023-12-16T07:46:00.028070+0000 mon.a (mon.0) 47 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:00.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[64479]: audit 2023-12-16T07:46:00.036365+0000 mon.a (mon.0) 48 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:00.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[64479]: audit 2023-12-16T07:46:00.042432+0000 mon.a (mon.0) 49 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:00.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:00 smithi007 bash[64479]: audit 2023-12-16T07:46:00.050174+0000 mon.a (mon.0) 50 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:01.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:01 smithi169 bash[25083]: audit 2023-12-16T07:45:59.767070+0000 mgr.y (mgr.25477) 13 : audit [DBG] from='client.25531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:01.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:01 smithi169 bash[25083]: audit 2023-12-16T07:46:00.261181+0000 mon.a (mon.0) 51 : audit [DBG] from='client.? 172.21.15.7:0/3030193102' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:46:01.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:01 smithi007 bash[64479]: audit 2023-12-16T07:45:59.767070+0000 mgr.y (mgr.25477) 13 : audit 0 from='client.25531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:01.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:01 smithi007 bash[64479]: audit 2023-12-16T07:46:00.261181+0000 mon.a (mon.0) 51 : audit 0 from='client.? 172.21.15.7:0/3030193102' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:46:01.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:01 smithi007 bash[30436]: audit 2023-12-16T07:45:59.767070+0000 mgr.y (mgr.25477) 13 : audit [DBG] from='client.25531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:01.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:01 smithi007 bash[30436]: audit 2023-12-16T07:46:00.261181+0000 mon.a (mon.0) 51 : audit [DBG] from='client.? 172.21.15.7:0/3030193102' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:46:02.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:02 smithi007 bash[30436]: cluster 2023-12-16T07:46:01.084376+0000 mgr.y (mgr.25477) 14 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2023-12-16T07:46:02.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:02 smithi007 bash[30436]: audit 2023-12-16T07:46:01.192220+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:02.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:02 smithi007 bash[30436]: audit 2023-12-16T07:46:01.201780+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:02.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:02 smithi007 bash[30436]: audit 2023-12-16T07:46:01.209315+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:02.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:02 smithi007 bash[30436]: audit 2023-12-16T07:46:01.216364+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:02.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:02 smithi007 bash[64479]: cluster 2023-12-16T07:46:01.084376+0000 mgr.y (mgr.25477) 14 : cluster 0 pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2023-12-16T07:46:02.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:02 smithi007 bash[64479]: audit 2023-12-16T07:46:01.192220+0000 mon.a (mon.0) 52 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:02.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:02 smithi007 bash[64479]: audit 2023-12-16T07:46:01.201780+0000 mon.a (mon.0) 53 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:02.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:02 smithi007 bash[64479]: audit 2023-12-16T07:46:01.209315+0000 mon.a (mon.0) 54 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:02.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:02 smithi007 bash[64479]: audit 2023-12-16T07:46:01.216364+0000 mon.a (mon.0) 55 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:02.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:02 smithi169 bash[25083]: cluster 2023-12-16T07:46:01.084376+0000 mgr.y (mgr.25477) 14 : cluster [DBG] pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2023-12-16T07:46:02.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:02 smithi169 bash[25083]: audit 2023-12-16T07:46:01.192220+0000 mon.a (mon.0) 52 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:02.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:02 smithi169 bash[25083]: audit 2023-12-16T07:46:01.201780+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:02.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:02 smithi169 bash[25083]: audit 2023-12-16T07:46:01.209315+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:02.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:02 smithi169 bash[25083]: audit 2023-12-16T07:46:01.216364+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:04.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:04 smithi007 bash[64479]: cluster 2023-12-16T07:46:03.085203+0000 mgr.y (mgr.25477) 15 : cluster 0 pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2023-12-16T07:46:04.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:04 smithi007 bash[30436]: cluster 2023-12-16T07:46:03.085203+0000 mgr.y (mgr.25477) 15 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2023-12-16T07:46:04.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:04 smithi169 bash[25083]: cluster 2023-12-16T07:46:03.085203+0000 mgr.y (mgr.25477) 15 : cluster [DBG] pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2023-12-16T07:46:05.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:05 smithi007 bash[64479]: audit 2023-12-16T07:46:04.091830+0000 mgr.y (mgr.25477) 16 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:05.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:05 smithi007 bash[30436]: audit 2023-12-16T07:46:04.091830+0000 mgr.y (mgr.25477) 16 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:05.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:05 smithi169 bash[25083]: audit 2023-12-16T07:46:04.091830+0000 mgr.y (mgr.25477) 16 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:06.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:06 smithi007 bash[30436]: cluster 2023-12-16T07:46:05.086978+0000 mgr.y (mgr.25477) 17 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2023-12-16T07:46:06.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:06 smithi007 bash[64479]: cluster 2023-12-16T07:46:05.086978+0000 mgr.y (mgr.25477) 17 : cluster 0 pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2023-12-16T07:46:06.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:06 smithi169 bash[25083]: cluster 2023-12-16T07:46:05.086978+0000 mgr.y (mgr.25477) 17 : cluster [DBG] pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2023-12-16T07:46:08.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:08 smithi007 bash[64479]: cluster 2023-12-16T07:46:07.087927+0000 mgr.y (mgr.25477) 18 : cluster 0 pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:46:08.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:08 smithi007 bash[64479]: audit 2023-12-16T07:46:08.197213+0000 mon.a (mon.0) 56 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:46:08.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:08 smithi007 bash[30436]: cluster 2023-12-16T07:46:07.087927+0000 mgr.y (mgr.25477) 18 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:46:08.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:08 smithi007 bash[30436]: audit 2023-12-16T07:46:08.197213+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:46:08.599 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:08 smithi169 bash[25083]: cluster 2023-12-16T07:46:07.087927+0000 mgr.y (mgr.25477) 18 : cluster [DBG] pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:46:08.599 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:08 smithi169 bash[25083]: audit 2023-12-16T07:46:08.197213+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:46:08.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:46:08 smithi169 bash[49087]: ts=2023-12-16T07:46:08.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:46:10.066 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:09 smithi169 bash[25083]: cluster 2023-12-16T07:46:09.089621+0000 mgr.y (mgr.25477) 19 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:46:10.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:09 smithi007 bash[30436]: cluster 2023-12-16T07:46:09.089621+0000 mgr.y (mgr.25477) 19 : cluster [DBG] pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:46:10.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:09 smithi007 bash[64479]: cluster 2023-12-16T07:46:09.089621+0000 mgr.y (mgr.25477) 19 : cluster 0 pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:46:10.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:46:10 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:46:10] "GET /metrics HTTP/1.1" 200 38159 "" "Prometheus/2.33.4" 2023-12-16T07:46:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: audit 2023-12-16T07:46:10.652888+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: audit 2023-12-16T07:46:10.663710+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:11.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: audit 2023-12-16T07:46:10.666780+0000 mon.a (mon.0) 59 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:46:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: audit 2023-12-16T07:46:10.668289+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:46:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: cephadm 2023-12-16T07:46:10.670595+0000 mgr.y (mgr.25477) 20 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:46:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: cephadm 2023-12-16T07:46:10.671109+0000 mgr.y (mgr.25477) 21 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:46:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: cephadm 2023-12-16T07:46:10.886193+0000 mgr.y (mgr.25477) 22 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:46:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: cephadm 2023-12-16T07:46:10.886583+0000 mgr.y (mgr.25477) 23 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:46:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: cluster 2023-12-16T07:46:11.090445+0000 mgr.y (mgr.25477) 24 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:46:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: cephadm 2023-12-16T07:46:11.102039+0000 mgr.y (mgr.25477) 25 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:46:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: cephadm 2023-12-16T07:46:11.102520+0000 mgr.y (mgr.25477) 26 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:46:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: cephadm 2023-12-16T07:46:11.317736+0000 mgr.y (mgr.25477) 27 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:46:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: cephadm 2023-12-16T07:46:11.318289+0000 mgr.y (mgr.25477) 28 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:46:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: audit 2023-12-16T07:46:11.491187+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: audit 2023-12-16T07:46:11.502701+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: audit 2023-12-16T07:46:11.541877+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:11.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: audit 2023-12-16T07:46:11.552227+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:11.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:11 smithi169 bash[25083]: audit 2023-12-16T07:46:11.561212+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: audit 2023-12-16T07:46:10.652888+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: audit 2023-12-16T07:46:10.663710+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: audit 2023-12-16T07:46:10.666780+0000 mon.a (mon.0) 59 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:46:12.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: audit 2023-12-16T07:46:10.668289+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:46:12.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: cephadm 2023-12-16T07:46:10.670595+0000 mgr.y (mgr.25477) 20 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:46:12.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: cephadm 2023-12-16T07:46:10.671109+0000 mgr.y (mgr.25477) 21 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:46:12.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: cephadm 2023-12-16T07:46:10.886193+0000 mgr.y (mgr.25477) 22 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:46:12.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: cephadm 2023-12-16T07:46:10.886583+0000 mgr.y (mgr.25477) 23 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:46:12.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: cluster 2023-12-16T07:46:11.090445+0000 mgr.y (mgr.25477) 24 : cluster [DBG] pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:46:12.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: cephadm 2023-12-16T07:46:11.102039+0000 mgr.y (mgr.25477) 25 : cephadm [INF] Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:46:12.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: cephadm 2023-12-16T07:46:11.102520+0000 mgr.y (mgr.25477) 26 : cephadm [INF] Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:46:12.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: cephadm 2023-12-16T07:46:11.317736+0000 mgr.y (mgr.25477) 27 : cephadm [INF] Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:46:12.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: cephadm 2023-12-16T07:46:11.318289+0000 mgr.y (mgr.25477) 28 : cephadm [INF] Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:46:12.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: audit 2023-12-16T07:46:11.491187+0000 mon.a (mon.0) 61 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: audit 2023-12-16T07:46:11.502701+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: audit 2023-12-16T07:46:11.541877+0000 mon.a (mon.0) 63 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: audit 2023-12-16T07:46:11.552227+0000 mon.a (mon.0) 64 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[30436]: audit 2023-12-16T07:46:11.561212+0000 mon.a (mon.0) 65 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: audit 2023-12-16T07:46:10.652888+0000 mon.a (mon.0) 57 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: audit 2023-12-16T07:46:10.663710+0000 mon.a (mon.0) 58 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: audit 2023-12-16T07:46:10.666780+0000 mon.a (mon.0) 59 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:46:12.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: audit 2023-12-16T07:46:10.668289+0000 mon.a (mon.0) 60 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:46:12.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: cephadm 2023-12-16T07:46:10.670595+0000 mgr.y (mgr.25477) 20 : cephadm 1 Updating smithi007:/etc/ceph/ceph.conf 2023-12-16T07:46:12.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: cephadm 2023-12-16T07:46:10.671109+0000 mgr.y (mgr.25477) 21 : cephadm 1 Updating smithi169:/etc/ceph/ceph.conf 2023-12-16T07:46:12.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: cephadm 2023-12-16T07:46:10.886193+0000 mgr.y (mgr.25477) 22 : cephadm 1 Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:46:12.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: cephadm 2023-12-16T07:46:10.886583+0000 mgr.y (mgr.25477) 23 : cephadm 1 Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.conf 2023-12-16T07:46:12.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: cluster 2023-12-16T07:46:11.090445+0000 mgr.y (mgr.25477) 24 : cluster 0 pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2023-12-16T07:46:12.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: cephadm 2023-12-16T07:46:11.102039+0000 mgr.y (mgr.25477) 25 : cephadm 1 Updating smithi169:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:46:12.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: cephadm 2023-12-16T07:46:11.102520+0000 mgr.y (mgr.25477) 26 : cephadm 1 Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2023-12-16T07:46:12.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: cephadm 2023-12-16T07:46:11.317736+0000 mgr.y (mgr.25477) 27 : cephadm 1 Updating smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:46:12.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: cephadm 2023-12-16T07:46:11.318289+0000 mgr.y (mgr.25477) 28 : cephadm 1 Updating smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/config/ceph.client.admin.keyring 2023-12-16T07:46:12.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: audit 2023-12-16T07:46:11.491187+0000 mon.a (mon.0) 61 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: audit 2023-12-16T07:46:11.502701+0000 mon.a (mon.0) 62 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: audit 2023-12-16T07:46:11.541877+0000 mon.a (mon.0) 63 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: audit 2023-12-16T07:46:11.552227+0000 mon.a (mon.0) 64 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:11 smithi007 bash[64479]: audit 2023-12-16T07:46:11.561212+0000 mon.a (mon.0) 65 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:12.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:12 smithi169 bash[25083]: audit 2023-12-16T07:46:11.650577+0000 mon.a (mon.0) 66 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:46:12.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:12 smithi169 bash[25083]: audit 2023-12-16T07:46:11.660925+0000 mon.a (mon.0) 67 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:46:12.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:12 smithi169 bash[25083]: audit 2023-12-16T07:46:11.666805+0000 mon.a (mon.0) 68 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-12-16T07:46:12.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:12 smithi169 bash[25083]: audit 2023-12-16T07:46:11.668003+0000 mon.a (mon.0) 69 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2023-12-16T07:46:12.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:12 smithi169 bash[25083]: cephadm 2023-12-16T07:46:11.668939+0000 mgr.y (mgr.25477) 29 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2023-12-16T07:46:13.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:12 smithi007 bash[30436]: audit 2023-12-16T07:46:11.650577+0000 mon.a (mon.0) 66 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:46:13.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:12 smithi007 bash[30436]: audit 2023-12-16T07:46:11.660925+0000 mon.a (mon.0) 67 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:46:13.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:12 smithi007 bash[30436]: audit 2023-12-16T07:46:11.666805+0000 mon.a (mon.0) 68 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-12-16T07:46:13.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:12 smithi007 bash[30436]: audit 2023-12-16T07:46:11.668003+0000 mon.a (mon.0) 69 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2023-12-16T07:46:13.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:12 smithi007 bash[30436]: cephadm 2023-12-16T07:46:11.668939+0000 mgr.y (mgr.25477) 29 : cephadm [INF] Upgrade: It appears safe to stop mon.c 2023-12-16T07:46:13.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:12 smithi007 bash[64479]: audit 2023-12-16T07:46:11.650577+0000 mon.a (mon.0) 66 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:46:13.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:12 smithi007 bash[64479]: audit 2023-12-16T07:46:11.660925+0000 mon.a (mon.0) 67 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:46:13.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:12 smithi007 bash[64479]: audit 2023-12-16T07:46:11.666805+0000 mon.a (mon.0) 68 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-12-16T07:46:13.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:12 smithi007 bash[64479]: audit 2023-12-16T07:46:11.668003+0000 mon.a (mon.0) 69 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2023-12-16T07:46:13.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:12 smithi007 bash[64479]: cephadm 2023-12-16T07:46:11.668939+0000 mgr.y (mgr.25477) 29 : cephadm 1 Upgrade: It appears safe to stop mon.c 2023-12-16T07:46:13.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:13 smithi007 bash[64479]: cluster 2023-12-16T07:46:13.091221+0000 mgr.y (mgr.25477) 30 : cluster 0 pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:13.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:13 smithi007 bash[30436]: cluster 2023-12-16T07:46:13.091221+0000 mgr.y (mgr.25477) 30 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:13.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:13 smithi169 bash[25083]: cluster 2023-12-16T07:46:13.091221+0000 mgr.y (mgr.25477) 30 : cluster [DBG] pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:15 smithi169 bash[25083]: audit 2023-12-16T07:46:14.099250+0000 mgr.y (mgr.25477) 31 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:15 smithi169 bash[25083]: cephadm 2023-12-16T07:46:14.339945+0000 mgr.y (mgr.25477) 32 : cephadm [INF] Upgrade: Updating mon.c 2023-12-16T07:46:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:15 smithi169 bash[25083]: audit 2023-12-16T07:46:14.347311+0000 mon.a (mon.0) 70 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:15 smithi169 bash[25083]: audit 2023-12-16T07:46:14.350979+0000 mon.a (mon.0) 71 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:46:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:15 smithi169 bash[25083]: audit 2023-12-16T07:46:14.352601+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:46:15.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:15 smithi169 bash[25083]: audit 2023-12-16T07:46:14.354255+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:46:15.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:15 smithi169 bash[25083]: cephadm 2023-12-16T07:46:14.356295+0000 mgr.y (mgr.25477) 33 : cephadm [INF] Deploying daemon mon.c on smithi007 2023-12-16T07:46:15.802 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:15 smithi007 bash[64479]: audit 2023-12-16T07:46:14.099250+0000 mgr.y (mgr.25477) 31 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:15.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:15 smithi007 bash[64479]: cephadm 2023-12-16T07:46:14.339945+0000 mgr.y (mgr.25477) 32 : cephadm 1 Upgrade: Updating mon.c 2023-12-16T07:46:15.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:15 smithi007 bash[64479]: audit 2023-12-16T07:46:14.347311+0000 mon.a (mon.0) 70 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:15.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:15 smithi007 bash[64479]: audit 2023-12-16T07:46:14.350979+0000 mon.a (mon.0) 71 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:46:15.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:15 smithi007 bash[64479]: audit 2023-12-16T07:46:14.352601+0000 mon.a (mon.0) 72 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:46:15.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:15 smithi007 bash[64479]: audit 2023-12-16T07:46:14.354255+0000 mon.a (mon.0) 73 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:46:15.803 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:15 smithi007 bash[64479]: cephadm 2023-12-16T07:46:14.356295+0000 mgr.y (mgr.25477) 33 : cephadm 1 Deploying daemon mon.c on smithi007 2023-12-16T07:46:15.803 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:15 smithi007 bash[30436]: audit 2023-12-16T07:46:14.099250+0000 mgr.y (mgr.25477) 31 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:15.803 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:15 smithi007 bash[30436]: cephadm 2023-12-16T07:46:14.339945+0000 mgr.y (mgr.25477) 32 : cephadm [INF] Upgrade: Updating mon.c 2023-12-16T07:46:15.804 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:15 smithi007 bash[30436]: audit 2023-12-16T07:46:14.347311+0000 mon.a (mon.0) 70 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:15.804 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:15 smithi007 bash[30436]: audit 2023-12-16T07:46:14.350979+0000 mon.a (mon.0) 71 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:46:15.804 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:15 smithi007 bash[30436]: audit 2023-12-16T07:46:14.352601+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:46:15.804 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:15 smithi007 bash[30436]: audit 2023-12-16T07:46:14.354255+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:46:15.804 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:15 smithi007 bash[30436]: cephadm 2023-12-16T07:46:14.356295+0000 mgr.y (mgr.25477) 33 : cephadm [INF] Deploying daemon mon.c on smithi007 2023-12-16T07:46:16.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:16 smithi007 bash[64479]: cluster 2023-12-16T07:46:15.092906+0000 mgr.y (mgr.25477) 34 : cluster 0 pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:16.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:16 smithi007 bash[30436]: cluster 2023-12-16T07:46:15.092906+0000 mgr.y (mgr.25477) 34 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:16.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:16 smithi169 bash[25083]: cluster 2023-12-16T07:46:15.092906+0000 mgr.y (mgr.25477) 34 : cluster [DBG] pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:17.279 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:17 smithi007 systemd[1]: Stopping Ceph mon.c for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:46:17.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:17 smithi007 bash[30436]: debug 2023-12-16T07:46:17.271+0000 7fe9dd978700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-16T07:46:17.557 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:17 smithi007 bash[30436]: debug 2023-12-16T07:46:17.271+0000 7fe9dd978700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-12-16T07:46:18.438 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:18 smithi007 bash[65711]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mon-c 2023-12-16T07:46:18.439 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:18 smithi007 bash[65797]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mon-c 2023-12-16T07:46:18.439 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:18 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.c.service: Succeeded. 2023-12-16T07:46:18.439 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:18 smithi007 systemd[1]: Stopped Ceph mon.c for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:46:18.729 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:18 smithi007 systemd[1]: Started Ceph mon.c for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:46:18.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:46:18 smithi169 bash[49087]: ts=2023-12-16T07:46:18.662Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:46:20.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.887+0000 7f724a079c80 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-16T07:46:20.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.887+0000 7f724a079c80 0 ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev), process ceph-mon, pid 7 2023-12-16T07:46:20.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.887+0000 7f724a079c80 0 pidfile_write: ignore empty --pid-file 2023-12-16T07:46:20.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 0 load: jerasure load: lrc 2023-12-16T07:46:20.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: RocksDB version: 7.9.2 2023-12-16T07:46:20.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Git sha 0 2023-12-16T07:46:20.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Compile date 2023-12-15 16:32:04 2023-12-16T07:46:20.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: DB SUMMARY 2023-12-16T07:46:20.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: DB Session ID: D4AIEITBLMZLFBCJ8GS4 2023-12-16T07:46:20.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: CURRENT file: CURRENT 2023-12-16T07:46:20.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: IDENTITY file: IDENTITY 2023-12-16T07:46:20.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: MANIFEST file: MANIFEST-000009 size: 1263 Bytes 2023-12-16T07:46:20.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000030.sst 2023-12-16T07:46:20.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000028.log size: 4129646 ; 2023-12-16T07:46:20.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.error_if_exists: 0 2023-12-16T07:46:20.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.create_if_missing: 0 2023-12-16T07:46:20.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.paranoid_checks: 1 2023-12-16T07:46:20.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.flush_verify_memtable_count: 1 2023-12-16T07:46:20.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-12-16T07:46:20.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-12-16T07:46:20.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.env: 0x55c8c2e0ea60 2023-12-16T07:46:20.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.fs: PosixFileSystem 2023-12-16T07:46:20.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.info_log: 0x55c8c53857a0 2023-12-16T07:46:20.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-16T07:46:20.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.statistics: (nil) 2023-12-16T07:46:20.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.use_fsync: 0 2023-12-16T07:46:20.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.max_log_file_size: 0 2023-12-16T07:46:20.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-16T07:46:20.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-16T07:46:20.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-16T07:46:20.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-16T07:46:20.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.allow_fallocate: 1 2023-12-16T07:46:20.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-16T07:46:20.070 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.use_direct_reads: 0 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.create_missing_column_families: 0 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.db_log_dir: 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.wal_dir: 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.advise_random_on_open: 1 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.write_buffer_manager: 0x55c8c53592c0 2023-12-16T07:46:20.071 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-16T07:46:20.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-16T07:46:20.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-16T07:46:20.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.rate_limiter: (nil) 2023-12-16T07:46:20.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-16T07:46:20.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-16T07:46:20.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-16T07:46:20.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-16T07:46:20.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.unordered_write: 0 2023-12-16T07:46:20.073 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-16T07:46:20.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-16T07:46:20.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-16T07:46:20.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-16T07:46:20.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.row_cache: None 2023-12-16T07:46:20.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.wal_filter: None 2023-12-16T07:46:20.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-16T07:46:20.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-16T07:46:20.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.two_write_queues: 0 2023-12-16T07:46:20.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.manual_wal_flush: 0 2023-12-16T07:46:20.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.wal_compression: 0 2023-12-16T07:46:20.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.atomic_flush: 0 2023-12-16T07:46:20.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-16T07:46:20.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-16T07:46:20.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.log_readahead_size: 0 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.best_efforts_recovery: 0 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.allow_data_in_errors: 0 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.db_host_id: __hostname__ 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.enforce_single_del_contracts: true 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.max_background_jobs: 2 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.max_background_compactions: -1 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.max_subcompactions: 1 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-16T07:46:20.075 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.max_total_wal_size: 0 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.max_open_files: -1 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.bytes_per_sync: 0 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.max_background_flushes: -1 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Compression algorithms supported: 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: kZSTD supported: 0 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: kXpressCompression supported: 0 2023-12-16T07:46:20.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-16T07:46:20.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: kZlibCompression supported: 1 2023-12-16T07:46:20.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: kSnappyCompression supported: 1 2023-12-16T07:46:20.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: kLZ4Compression supported: 1 2023-12-16T07:46:20.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: kBZip2Compression supported: 0 2023-12-16T07:46:20.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-16T07:46:20.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: DMutex implementation: pthread_mutex_t 2023-12-16T07:46:20.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2023-12-16T07:46:20.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-12-16T07:46:20.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-16T07:46:20.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.merge_operator: 2023-12-16T07:46:20.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.compaction_filter: None 2023-12-16T07:46:20.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.compaction_filter_factory: None 2023-12-16T07:46:20.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.sst_partitioner_factory: None 2023-12-16T07:46:20.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c8c53858c0) 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: cache_index_and_filter_blocks: 1 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: pin_top_level_index_and_filter: 1 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: index_type: 0 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: data_block_index_type: 0 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: index_shortening: 1 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: data_block_hash_table_util_ratio: 0.750000 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: checksum: 4 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: no_block_cache: 0 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: block_cache: 0x55c8c53d9090 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: block_cache_name: BinnedLRUCache 2023-12-16T07:46:20.078 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: block_cache_options: 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: capacity : 536870912 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: num_shard_bits : 4 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: strict_capacity_limit : 0 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: high_pri_pool_ratio: 0.000 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: block_cache_compressed: (nil) 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: persistent_cache: (nil) 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: block_size: 4096 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: block_size_deviation: 10 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: block_restart_interval: 16 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: index_block_restart_interval: 1 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: metadata_block_size: 4096 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: partition_filters: 0 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: use_delta_encoding: 1 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: filter_policy: bloomfilter 2023-12-16T07:46:20.079 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: whole_key_filtering: 1 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: verify_compression: 0 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: read_amp_bytes_per_bit: 0 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: format_version: 5 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: enable_index_compression: 1 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: block_align: 0 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: max_auto_readahead_size: 262144 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: prepopulate_block_cache: 0 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: initial_auto_readahead_size: 8192 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: num_file_reads_for_auto_readahead: 2 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.compression: NoCompression 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-16T07:46:20.080 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-16T07:46:20.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.num_levels: 7 2023-12-16T07:46:20.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-16T07:46:20.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-16T07:46:20.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-16T07:46:20.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-16T07:46:20.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-16T07:46:20.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-16T07:46:20.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-16T07:46:20.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-16T07:46:20.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-12-16T07:46:20.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-16T07:46:20.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-12-16T07:46:20.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-12-16T07:46:20.081 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.compression_opts.level: 32767 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.891+0000 7f724a079c80 4 rocksdb: Options.compression_opts.enabled: false 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-16T07:46:20.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.arena_block_size: 1048576 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-16T07:46:20.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-16T07:46:20.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-16T07:46:20.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-16T07:46:20.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-16T07:46:20.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-16T07:46:20.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-16T07:46:20.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-16T07:46:20.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-16T07:46:20.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-16T07:46:20.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-16T07:46:20.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-16T07:46:20.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.table_properties_collectors: 2023-12-16T07:46:20.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.inplace_update_support: 0 2023-12-16T07:46:20.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-16T07:46:20.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.bloom_locality: 0 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.max_successive_merges: 0 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.force_consistency_checks: 1 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.ttl: 2592000 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.enable_blob_files: false 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.min_blob_size: 0 2023-12-16T07:46:20.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.blob_file_size: 268435456 2023-12-16T07:46:20.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.blob_compression_type: NoCompression 2023-12-16T07:46:20.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-12-16T07:46:20.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-12-16T07:46:20.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-12-16T07:46:20.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2023-12-16T07:46:20.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.blob_file_starting_level: 0 2023-12-16T07:46:20.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-12-16T07:46:20.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f723f841700 3 rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2023-12-16T07:46:20.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-12-16T07:46:20.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 32, last_sequence is 18092, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-16T07:46:20.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2023-12-16T07:46:20.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ae593167-e7ee-44bc-a4a6-2e1aa089ddc6 2023-12-16T07:46:20.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702712779900569, "job": 1, "event": "recovery_started", "wal_files": [28]} 2023-12-16T07:46:20.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.895+0000 7f724a079c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2023-12-16T07:46:20.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.911+0000 7f724a079c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702712779916799, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 2659955, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 18097, "largest_seqno": 19012, "table_properties": {"data_size": 2655536, "index_size": 2377, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1029, "raw_key_size": 9251, "raw_average_key_size": 24, "raw_value_size": 2647281, "raw_average_value_size": 6876, "num_data_blocks": 107, "num_entries": 385, "num_filter_entries": 385, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1702712779, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ae593167-e7ee-44bc-a4a6-2e1aa089ddc6", "db_session_id": "D4AIEITBLMZLFBCJ8GS4", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2023-12-16T07:46:20.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.911+0000 7f724a079c80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702712779916919, "job": 1, "event": "recovery_finished"} 2023-12-16T07:46:20.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.911+0000 7f724a079c80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 35 2023-12-16T07:46:20.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.911+0000 7f724a079c80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-12-16T07:46:20.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.915+0000 7f724a079c80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-12-16T07:46:20.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.915+0000 7f724a079c80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c8c5478000 2023-12-16T07:46:20.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.915+0000 7f724a079c80 4 rocksdb: DB pointer 0x55c8c5466000 2023-12-16T07:46:20.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.915+0000 7f7238833700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-12-16T07:46:20.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.915+0000 7f7238833700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2023-12-16T07:46:20.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: ** DB Stats ** 2023-12-16T07:46:20.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:46:20.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-16T07:46:20.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-16T07:46:20.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: ** Compaction Stats [default] ** 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: L0 1/0 2.54 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 394.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: L6 1/0 9.93 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Sum 2/0 12.46 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 394.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 394.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: ** Compaction Stats [default] ** 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 394.9 0.01 0.00 1 0.006 0 0 0.0 0.0 2023-12-16T07:46:20.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-12-16T07:46:20.089 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:46:20.089 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Flush(GB): cumulative 0.002, interval 0.002 2023-12-16T07:46:20.089 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T07:46:20.089 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T07:46:20.089 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T07:46:20.089 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: AddFile(Keys): cumulative 0, interval 0 2023-12-16T07:46:20.089 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Cumulative compaction: 0.00 GB write, 114.17 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:46:20.089 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Interval compaction: 0.00 GB write, 114.17 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:46:20.089 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T07:46:20.089 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Block cache BinnedLRUCache@0x55c8c53d9090#7 capacity: 512.00 MB usage: 3.73 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2023-12-16T07:46:20.089 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: Block cache entry stats(count,size,portion): FilterBlock(1,1.16 KB,0.000220537%) IndexBlock(1,2.58 KB,0.000491738%) Misc(1,0.00 KB,0%) 2023-12-16T07:46:20.089 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: ** File Read Latency Histogram By Level [default] ** 2023-12-16T07:46:20.089 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.915+0000 7f724a079c80 0 starting mon.c rank 1 at public addrs [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] at bind addrs [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:46:20.089 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.919+0000 7f724a079c80 1 mon.c@-1(???) e3 preinit fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:46:20.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.919+0000 7f724a079c80 0 mon.c@-1(???).mds e1 new map 2023-12-16T07:46:20.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.919+0000 7f724a079c80 0 mon.c@-1(???).mds e1 print_map 2023-12-16T07:46:20.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: e1 2023-12-16T07:46:20.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: enable_multiple, ever_enabled_multiple: 1,1 2023-12-16T07:46:20.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T07:46:20.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: legacy client fscid: -1 2023-12-16T07:46:20.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: 2023-12-16T07:46:20.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: No filesystems configured 2023-12-16T07:46:20.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.919+0000 7f724a079c80 0 mon.c@-1(???).osd e103 crush map has features 3314933000854323200, adjusting msgr requires 2023-12-16T07:46:20.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.919+0000 7f724a079c80 0 mon.c@-1(???).osd e103 crush map has features 432629239337189376, adjusting msgr requires 2023-12-16T07:46:20.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.919+0000 7f724a079c80 0 mon.c@-1(???).osd e103 crush map has features 432629239337189376, adjusting msgr requires 2023-12-16T07:46:20.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.919+0000 7f724a079c80 0 mon.c@-1(???).osd e103 crush map has features 432629239337189376, adjusting msgr requires 2023-12-16T07:46:20.090 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:19 smithi007 bash[65864]: debug 2023-12-16T07:46:19.919+0000 7f724a079c80 1 mon.c@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2023-12-16T07:46:20.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:46:20 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:46:20] "GET /metrics HTTP/1.1" 200 38159 "" "Prometheus/2.33.4" 2023-12-16T07:46:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:21 smithi169 bash[25083]: cluster 2023-12-16T07:46:19.094969+0000 mgr.y (mgr.25477) 36 : cluster [DBG] pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:21 smithi169 bash[25083]: cluster 2023-12-16T07:46:20.140801+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2023-12-16T07:46:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:21 smithi169 bash[25083]: cluster 2023-12-16T07:46:20.373229+0000 mon.a (mon.0) 74 : cluster [INF] mon.a calling monitor election 2023-12-16T07:46:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:21 smithi169 bash[25083]: cluster 2023-12-16T07:46:20.376312+0000 mon.a (mon.0) 75 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-12-16T07:46:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:21 smithi169 bash[25083]: cluster 2023-12-16T07:46:20.385316+0000 mon.a (mon.0) 76 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T07:46:21.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:21 smithi169 bash[25083]: cluster 2023-12-16T07:46:20.386146+0000 mon.a (mon.0) 77 : cluster [DBG] fsmap 2023-12-16T07:46:21.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:21 smithi169 bash[25083]: cluster 2023-12-16T07:46:20.386206+0000 mon.a (mon.0) 78 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2023-12-16T07:46:21.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:21 smithi169 bash[25083]: cluster 2023-12-16T07:46:20.386847+0000 mon.a (mon.0) 79 : cluster [DBG] mgrmap e40: y(active, since 27s), standbys: x 2023-12-16T07:46:21.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:21 smithi169 bash[25083]: cluster 2023-12-16T07:46:20.393963+0000 mon.a (mon.0) 80 : cluster [INF] overall HEALTH_OK 2023-12-16T07:46:21.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:21 smithi169 bash[25083]: audit 2023-12-16T07:46:20.398085+0000 mon.a (mon.0) 81 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:21.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:21 smithi169 bash[25083]: audit 2023-12-16T07:46:20.406273+0000 mon.a (mon.0) 82 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[65864]: cluster 2023-12-16T07:46:19.094969+0000 mgr.y (mgr.25477) 36 : cluster 0 pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[65864]: cluster 2023-12-16T07:46:20.140801+0000 mon.c (mon.1) 1 : cluster 1 mon.c calling monitor election 2023-12-16T07:46:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[65864]: cluster 2023-12-16T07:46:20.373229+0000 mon.a (mon.0) 74 : cluster 1 mon.a calling monitor election 2023-12-16T07:46:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[65864]: cluster 2023-12-16T07:46:20.376312+0000 mon.a (mon.0) 75 : cluster 1 mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-12-16T07:46:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[65864]: cluster 2023-12-16T07:46:20.385316+0000 mon.a (mon.0) 76 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T07:46:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[65864]: cluster 2023-12-16T07:46:20.386146+0000 mon.a (mon.0) 77 : cluster 0 fsmap 2023-12-16T07:46:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[65864]: cluster 2023-12-16T07:46:20.386206+0000 mon.a (mon.0) 78 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2023-12-16T07:46:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[65864]: cluster 2023-12-16T07:46:20.386847+0000 mon.a (mon.0) 79 : cluster 0 mgrmap e40: y(active, since 27s), standbys: x 2023-12-16T07:46:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[65864]: cluster 2023-12-16T07:46:20.393963+0000 mon.a (mon.0) 80 : cluster 1 overall HEALTH_OK 2023-12-16T07:46:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[65864]: audit 2023-12-16T07:46:20.398085+0000 mon.a (mon.0) 81 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:21.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[65864]: audit 2023-12-16T07:46:20.406273+0000 mon.a (mon.0) 82 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:21.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[64479]: cluster 2023-12-16T07:46:19.094969+0000 mgr.y (mgr.25477) 36 : cluster 0 pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:21.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[64479]: cluster 2023-12-16T07:46:20.140801+0000 mon.c (mon.1) 1 : cluster 1 mon.c calling monitor election 2023-12-16T07:46:21.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[64479]: cluster 2023-12-16T07:46:20.373229+0000 mon.a (mon.0) 74 : cluster 1 mon.a calling monitor election 2023-12-16T07:46:21.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[64479]: cluster 2023-12-16T07:46:20.376312+0000 mon.a (mon.0) 75 : cluster 1 mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-12-16T07:46:21.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[64479]: cluster 2023-12-16T07:46:20.385316+0000 mon.a (mon.0) 76 : cluster 0 monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T07:46:21.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[64479]: cluster 2023-12-16T07:46:20.386146+0000 mon.a (mon.0) 77 : cluster 0 fsmap 2023-12-16T07:46:21.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[64479]: cluster 2023-12-16T07:46:20.386206+0000 mon.a (mon.0) 78 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2023-12-16T07:46:21.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[64479]: cluster 2023-12-16T07:46:20.386847+0000 mon.a (mon.0) 79 : cluster 0 mgrmap e40: y(active, since 27s), standbys: x 2023-12-16T07:46:21.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[64479]: cluster 2023-12-16T07:46:20.393963+0000 mon.a (mon.0) 80 : cluster 1 overall HEALTH_OK 2023-12-16T07:46:21.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[64479]: audit 2023-12-16T07:46:20.398085+0000 mon.a (mon.0) 81 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:21.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:21 smithi007 bash[64479]: audit 2023-12-16T07:46:20.406273+0000 mon.a (mon.0) 82 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:22 smithi169 bash[25083]: cluster 2023-12-16T07:46:21.095740+0000 mgr.y (mgr.25477) 37 : cluster [DBG] pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:22 smithi007 bash[64479]: cluster 2023-12-16T07:46:21.095740+0000 mgr.y (mgr.25477) 37 : cluster 0 pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:22 smithi007 bash[65864]: cluster 2023-12-16T07:46:21.095740+0000 mgr.y (mgr.25477) 37 : cluster 0 pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:23.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:23 smithi169 bash[25083]: audit 2023-12-16T07:46:23.197264+0000 mon.a (mon.0) 83 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:46:23.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:23 smithi007 bash[64479]: audit 2023-12-16T07:46:23.197264+0000 mon.a (mon.0) 83 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:46:23.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:23 smithi007 bash[65864]: audit 2023-12-16T07:46:23.197264+0000 mon.a (mon.0) 83 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:46:24.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:24 smithi169 bash[25083]: cluster 2023-12-16T07:46:23.096492+0000 mgr.y (mgr.25477) 38 : cluster [DBG] pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:24.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:24 smithi007 bash[64479]: cluster 2023-12-16T07:46:23.096492+0000 mgr.y (mgr.25477) 38 : cluster 0 pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:24.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:24 smithi007 bash[65864]: cluster 2023-12-16T07:46:23.096492+0000 mgr.y (mgr.25477) 38 : cluster 0 pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:25.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:46:24 smithi007 bash[61260]: debug 2023-12-16T07:46:24.931+0000 7f2cea7a7700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-16T07:46:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:25 smithi169 bash[25083]: audit 2023-12-16T07:46:24.110544+0000 mgr.y (mgr.25477) 39 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:25.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:25 smithi007 bash[65864]: audit 2023-12-16T07:46:24.110544+0000 mgr.y (mgr.25477) 39 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:25.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:25 smithi007 bash[64479]: audit 2023-12-16T07:46:24.110544+0000 mgr.y (mgr.25477) 39 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:26.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:26 smithi169 bash[25083]: cluster 2023-12-16T07:46:25.098225+0000 mgr.y (mgr.25477) 40 : cluster [DBG] pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:26.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:26 smithi007 bash[65864]: cluster 2023-12-16T07:46:25.098225+0000 mgr.y (mgr.25477) 40 : cluster 0 pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:26.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:26 smithi007 bash[64479]: cluster 2023-12-16T07:46:25.098225+0000 mgr.y (mgr.25477) 40 : cluster 0 pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:28.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:27 smithi007 bash[65864]: audit 2023-12-16T07:46:26.982558+0000 mon.a (mon.0) 84 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:28.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:27 smithi007 bash[65864]: audit 2023-12-16T07:46:26.993520+0000 mon.a (mon.0) 85 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:28.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:27 smithi007 bash[65864]: cluster 2023-12-16T07:46:27.099113+0000 mgr.y (mgr.25477) 41 : cluster 0 pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:28.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:27 smithi007 bash[64479]: audit 2023-12-16T07:46:26.982558+0000 mon.a (mon.0) 84 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:28.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:27 smithi007 bash[64479]: audit 2023-12-16T07:46:26.993520+0000 mon.a (mon.0) 85 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:28.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:27 smithi007 bash[64479]: cluster 2023-12-16T07:46:27.099113+0000 mgr.y (mgr.25477) 41 : cluster 0 pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:28.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:27 smithi169 bash[25083]: audit 2023-12-16T07:46:26.982558+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:28.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:27 smithi169 bash[25083]: audit 2023-12-16T07:46:26.993520+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:28.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:27 smithi169 bash[25083]: cluster 2023-12-16T07:46:27.099113+0000 mgr.y (mgr.25477) 41 : cluster [DBG] pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:28.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:46:28 smithi169 bash[49087]: ts=2023-12-16T07:46:28.662Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:46:29.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:29 smithi007 bash[65864]: audit 2023-12-16T07:46:28.042598+0000 mon.a (mon.0) 86 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:29.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:29 smithi007 bash[65864]: audit 2023-12-16T07:46:28.053421+0000 mon.a (mon.0) 87 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:29.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:29 smithi007 bash[64479]: audit 2023-12-16T07:46:28.042598+0000 mon.a (mon.0) 86 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:29.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:29 smithi007 bash[64479]: audit 2023-12-16T07:46:28.053421+0000 mon.a (mon.0) 87 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:29.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:29 smithi169 bash[25083]: audit 2023-12-16T07:46:28.042598+0000 mon.a (mon.0) 86 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:29.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:29 smithi169 bash[25083]: audit 2023-12-16T07:46:28.053421+0000 mon.a (mon.0) 87 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:30.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:30 smithi007 bash[64479]: cluster 2023-12-16T07:46:29.100727+0000 mgr.y (mgr.25477) 42 : cluster 0 pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:30.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:46:30 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:46:30] "GET /metrics HTTP/1.1" 200 38094 "" "Prometheus/2.33.4" 2023-12-16T07:46:30.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:30 smithi007 bash[65864]: cluster 2023-12-16T07:46:29.100727+0000 mgr.y (mgr.25477) 42 : cluster 0 pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:30.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:30 smithi169 bash[25083]: cluster 2023-12-16T07:46:29.100727+0000 mgr.y (mgr.25477) 42 : cluster [DBG] pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:30.675 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:46:31.446 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:46:31.446 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (2m) 4s ago 10m 16.4M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:46:31.446 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (2m) 31s ago 9m 42.8M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:46:31.446 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (85s) 4s ago 7m 45.7M - 3.5 e1d6a67b021e 8d8009d0c83f 2023-12-16T07:46:31.446 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283,8765 running (73s) 31s ago 16m 431M - 19.0.0-191-g6d933094 a29c258b5d6f 5295ba35c2cd 2023-12-16T07:46:31.446 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (2m) 4s ago 18m 478M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:46:31.446 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (48s) 4s ago 18m 39.0M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 5a266b45f3ce 2023-12-16T07:46:31.446 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (16m) 31s ago 16m 57.3M 2048M 17.2.0 e1d6a67b021e b47a960c6102 2023-12-16T07:46:31.446 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (12s) 4s ago 16m 22.5M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1dc91c31fc27 2023-12-16T07:46:31.446 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (3m) 4s ago 10m 11.0M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:46:31.446 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (2m) 31s ago 10m 13.9M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:46:31.446 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (15m) 4s ago 15m 58.3M 2782M 17.2.0 e1d6a67b021e dbc4d9538238 2023-12-16T07:46:31.447 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (14m) 4s ago 15m 59.2M 2782M 17.2.0 e1d6a67b021e 14637e0f389c 2023-12-16T07:46:31.447 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (14m) 4s ago 14m 57.9M 2782M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:46:31.447 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (13m) 4s ago 13m 57.1M 2782M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:46:31.447 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (13m) 31s ago 13m 58.3M 3037M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:46:31.447 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (12m) 31s ago 12m 55.0M 3037M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:46:31.447 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (11m) 31s ago 11m 55.6M 3037M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:46:31.447 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (11m) 31s ago 11m 56.5M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:46:31.447 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (79s) 31s ago 10m 54.4M - 2.33.4 514e6a882f6e 2652ce1b8540 2023-12-16T07:46:31.447 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (8m) 4s ago 8m 87.0M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:46:31.447 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (8m) 31s ago 8m 87.0M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:46:31.447 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (8m) 4s ago 8m 86.8M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:46:31.447 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (7m) 31s ago 7m 86.7M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:46:31.904 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:46:31.904 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:46:31.904 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2023-12-16T07:46:31.904 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:46:31.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:46:31.904 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:46:31.904 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:46:31.904 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:46:31.904 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:46:31.904 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2023-12-16T07:46:31.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:46:31.905 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:46:31.905 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:46:31.905 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:46:31.905 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:46:31.905 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2023-12-16T07:46:31.905 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 4 2023-12-16T07:46:31.905 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:46:31.905 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:46:32.287 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:46:32.287 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2023-12-16T07:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2023-12-16T07:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "4/23 daemons upgraded", 2023-12-16T07:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading mon daemons", 2023-12-16T07:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:46:32.288 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:46:32.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:32 smithi169 bash[25083]: audit 2023-12-16T07:46:30.656869+0000 mgr.y (mgr.25477) 43 : audit [DBG] from='client.34167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:32.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:32 smithi169 bash[25083]: audit 2023-12-16T07:46:31.049504+0000 mgr.y (mgr.25477) 44 : audit [DBG] from='client.44107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:32.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:32 smithi169 bash[25083]: cluster 2023-12-16T07:46:31.101525+0000 mgr.y (mgr.25477) 45 : cluster [DBG] pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:32.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:32 smithi169 bash[25083]: audit 2023-12-16T07:46:31.418300+0000 mgr.y (mgr.25477) 46 : audit [DBG] from='client.44113 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:32.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:32 smithi169 bash[25083]: audit 2023-12-16T07:46:31.901405+0000 mon.a (mon.0) 88 : audit [DBG] from='client.? 172.21.15.7:0/929569992' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:46:32.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:32 smithi007 bash[64479]: audit 2023-12-16T07:46:30.656869+0000 mgr.y (mgr.25477) 43 : audit 0 from='client.34167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:32.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:32 smithi007 bash[64479]: audit 2023-12-16T07:46:31.049504+0000 mgr.y (mgr.25477) 44 : audit 0 from='client.44107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:32.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:32 smithi007 bash[64479]: cluster 2023-12-16T07:46:31.101525+0000 mgr.y (mgr.25477) 45 : cluster 0 pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:32.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:32 smithi007 bash[64479]: audit 2023-12-16T07:46:31.418300+0000 mgr.y (mgr.25477) 46 : audit 0 from='client.44113 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:32.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:32 smithi007 bash[64479]: audit 2023-12-16T07:46:31.901405+0000 mon.a (mon.0) 88 : audit 0 from='client.? 172.21.15.7:0/929569992' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:46:32.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:32 smithi007 bash[65864]: audit 2023-12-16T07:46:30.656869+0000 mgr.y (mgr.25477) 43 : audit 0 from='client.34167 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:32.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:32 smithi007 bash[65864]: audit 2023-12-16T07:46:31.049504+0000 mgr.y (mgr.25477) 44 : audit 0 from='client.44107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:32.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:32 smithi007 bash[65864]: cluster 2023-12-16T07:46:31.101525+0000 mgr.y (mgr.25477) 45 : cluster 0 pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:32.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:32 smithi007 bash[65864]: audit 2023-12-16T07:46:31.418300+0000 mgr.y (mgr.25477) 46 : audit 0 from='client.44113 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:32.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:32 smithi007 bash[65864]: audit 2023-12-16T07:46:31.901405+0000 mon.a (mon.0) 88 : audit 0 from='client.? 172.21.15.7:0/929569992' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:46:32.746 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:46:33.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:33 smithi007 bash[64479]: audit 2023-12-16T07:46:32.284581+0000 mgr.y (mgr.25477) 47 : audit 0 from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:33.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:33 smithi007 bash[64479]: audit 2023-12-16T07:46:32.743533+0000 mon.c (mon.1) 2 : audit 0 from='client.? 172.21.15.7:0/1320503308' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:46:33.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:33 smithi007 bash[65864]: audit 2023-12-16T07:46:32.284581+0000 mgr.y (mgr.25477) 47 : audit 0 from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:33.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:33 smithi007 bash[65864]: audit 2023-12-16T07:46:32.743533+0000 mon.c (mon.1) 2 : audit 0 from='client.? 172.21.15.7:0/1320503308' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:46:33.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:33 smithi169 bash[25083]: audit 2023-12-16T07:46:32.284581+0000 mgr.y (mgr.25477) 47 : audit [DBG] from='client.44125 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:46:33.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:33 smithi169 bash[25083]: audit 2023-12-16T07:46:32.743533+0000 mon.c (mon.1) 2 : audit [DBG] from='client.? 172.21.15.7:0/1320503308' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:46:34.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:34 smithi169 bash[25083]: cluster 2023-12-16T07:46:33.102251+0000 mgr.y (mgr.25477) 48 : cluster [DBG] pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:34.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:34 smithi007 bash[64479]: cluster 2023-12-16T07:46:33.102251+0000 mgr.y (mgr.25477) 48 : cluster 0 pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:34.460 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:34 smithi007 bash[65864]: cluster 2023-12-16T07:46:33.102251+0000 mgr.y (mgr.25477) 48 : cluster 0 pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:35.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:35 smithi169 bash[25083]: audit 2023-12-16T07:46:34.119238+0000 mgr.y (mgr.25477) 49 : audit [DBG] from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:35.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:35 smithi007 bash[64479]: audit 2023-12-16T07:46:34.119238+0000 mgr.y (mgr.25477) 49 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:35.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:35 smithi007 bash[65864]: audit 2023-12-16T07:46:34.119238+0000 mgr.y (mgr.25477) 49 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:36.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:36 smithi007 bash[65864]: cluster 2023-12-16T07:46:35.103967+0000 mgr.y (mgr.25477) 50 : cluster 0 pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:36.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:36 smithi007 bash[64479]: cluster 2023-12-16T07:46:35.103967+0000 mgr.y (mgr.25477) 50 : cluster 0 pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:36.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:36 smithi169 bash[25083]: cluster 2023-12-16T07:46:35.103967+0000 mgr.y (mgr.25477) 50 : cluster [DBG] pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:38.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:38 smithi007 bash[64479]: cluster 2023-12-16T07:46:37.104759+0000 mgr.y (mgr.25477) 51 : cluster 0 pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:39.571 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:38 smithi007 bash[65864]: cluster 2023-12-16T07:46:37.104759+0000 mgr.y (mgr.25477) 51 : cluster 0 pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:39.572 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[64479]: audit 2023-12-16T07:46:38.197473+0000 mon.a (mon.0) 89 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:46:39.572 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[64479]: audit 2023-12-16T07:46:38.458683+0000 mon.a (mon.0) 90 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:39.572 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[64479]: audit 2023-12-16T07:46:38.470290+0000 mon.a (mon.0) 91 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:39.572 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[64479]: audit 2023-12-16T07:46:38.472894+0000 mon.a (mon.0) 92 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:46:39.572 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[64479]: audit 2023-12-16T07:46:38.474441+0000 mon.a (mon.0) 93 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:46:39.572 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[64479]: audit 2023-12-16T07:46:38.482334+0000 mon.a (mon.0) 94 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:39.572 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[64479]: audit 2023-12-16T07:46:38.567664+0000 mon.a (mon.0) 95 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:46:39.572 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[64479]: audit 2023-12-16T07:46:38.571637+0000 mon.a (mon.0) 96 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:46:39.573 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[64479]: audit 2023-12-16T07:46:38.574258+0000 mon.a (mon.0) 97 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-12-16T07:46:39.573 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[64479]: audit 2023-12-16T07:46:38.575962+0000 mon.a (mon.0) 98 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2023-12-16T07:46:39.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[65864]: audit 2023-12-16T07:46:38.197473+0000 mon.a (mon.0) 89 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:46:39.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[65864]: audit 2023-12-16T07:46:38.458683+0000 mon.a (mon.0) 90 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:39.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[65864]: audit 2023-12-16T07:46:38.470290+0000 mon.a (mon.0) 91 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:39.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[65864]: audit 2023-12-16T07:46:38.472894+0000 mon.a (mon.0) 92 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:46:39.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[65864]: audit 2023-12-16T07:46:38.474441+0000 mon.a (mon.0) 93 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:46:39.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[65864]: audit 2023-12-16T07:46:38.482334+0000 mon.a (mon.0) 94 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:39.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[65864]: audit 2023-12-16T07:46:38.567664+0000 mon.a (mon.0) 95 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:46:39.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[65864]: audit 2023-12-16T07:46:38.571637+0000 mon.a (mon.0) 96 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:46:39.574 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[65864]: audit 2023-12-16T07:46:38.574258+0000 mon.a (mon.0) 97 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-12-16T07:46:39.574 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:39 smithi007 bash[65864]: audit 2023-12-16T07:46:38.575962+0000 mon.a (mon.0) 98 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2023-12-16T07:46:39.574 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:38 smithi169 bash[25083]: cluster 2023-12-16T07:46:37.104759+0000 mgr.y (mgr.25477) 51 : cluster [DBG] pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:39.574 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:46:38 smithi169 bash[49087]: ts=2023-12-16T07:46:38.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:46:39.575 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:39 smithi169 bash[25083]: audit 2023-12-16T07:46:38.197473+0000 mon.a (mon.0) 89 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:46:39.575 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:39 smithi169 bash[25083]: audit 2023-12-16T07:46:38.458683+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:39.575 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:39 smithi169 bash[25083]: audit 2023-12-16T07:46:38.470290+0000 mon.a (mon.0) 91 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:39.575 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:39 smithi169 bash[25083]: audit 2023-12-16T07:46:38.472894+0000 mon.a (mon.0) 92 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:46:39.575 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:39 smithi169 bash[25083]: audit 2023-12-16T07:46:38.474441+0000 mon.a (mon.0) 93 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:46:39.575 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:39 smithi169 bash[25083]: audit 2023-12-16T07:46:38.482334+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:39.575 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:39 smithi169 bash[25083]: audit 2023-12-16T07:46:38.567664+0000 mon.a (mon.0) 95 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:46:39.575 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:39 smithi169 bash[25083]: audit 2023-12-16T07:46:38.571637+0000 mon.a (mon.0) 96 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:46:39.575 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:39 smithi169 bash[25083]: audit 2023-12-16T07:46:38.574258+0000 mon.a (mon.0) 97 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2023-12-16T07:46:39.575 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:39 smithi169 bash[25083]: audit 2023-12-16T07:46:38.575962+0000 mon.a (mon.0) 98 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2023-12-16T07:46:40.344 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:40 smithi169 bash[25083]: cephadm 2023-12-16T07:46:38.577230+0000 mgr.y (mgr.25477) 52 : cephadm [INF] Upgrade: It appears safe to stop mon.b 2023-12-16T07:46:40.344 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:40 smithi169 bash[25083]: cluster 2023-12-16T07:46:39.106430+0000 mgr.y (mgr.25477) 53 : cluster [DBG] pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:40.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:46:40 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:46:40] "GET /metrics HTTP/1.1" 200 38164 "" "Prometheus/2.33.4" 2023-12-16T07:46:40.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:40 smithi007 bash[64479]: cephadm 2023-12-16T07:46:38.577230+0000 mgr.y (mgr.25477) 52 : cephadm 1 Upgrade: It appears safe to stop mon.b 2023-12-16T07:46:40.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:40 smithi007 bash[64479]: cluster 2023-12-16T07:46:39.106430+0000 mgr.y (mgr.25477) 53 : cluster 0 pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:40.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:40 smithi007 bash[65864]: cephadm 2023-12-16T07:46:38.577230+0000 mgr.y (mgr.25477) 52 : cephadm 1 Upgrade: It appears safe to stop mon.b 2023-12-16T07:46:40.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:40 smithi007 bash[65864]: cluster 2023-12-16T07:46:39.106430+0000 mgr.y (mgr.25477) 53 : cluster 0 pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:42.393 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:42 smithi169 bash[25083]: cephadm 2023-12-16T07:46:41.097762+0000 mgr.y (mgr.25477) 54 : cephadm [INF] Upgrade: Updating mon.b 2023-12-16T07:46:42.393 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:42 smithi169 bash[25083]: audit 2023-12-16T07:46:41.104982+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:42.393 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:42 smithi169 bash[25083]: cluster 2023-12-16T07:46:41.107229+0000 mgr.y (mgr.25477) 55 : cluster [DBG] pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:42.393 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:42 smithi169 bash[25083]: audit 2023-12-16T07:46:41.108945+0000 mon.a (mon.0) 100 : audit [INF] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:46:42.393 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:42 smithi169 bash[25083]: audit 2023-12-16T07:46:41.110319+0000 mon.a (mon.0) 101 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:46:42.393 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:42 smithi169 bash[25083]: audit 2023-12-16T07:46:41.111791+0000 mon.a (mon.0) 102 : audit [DBG] from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:46:42.394 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:42 smithi169 bash[25083]: cephadm 2023-12-16T07:46:41.113773+0000 mgr.y (mgr.25477) 56 : cephadm [INF] Deploying daemon mon.b on smithi169 2023-12-16T07:46:42.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:42 smithi007 bash[64479]: cephadm 2023-12-16T07:46:41.097762+0000 mgr.y (mgr.25477) 54 : cephadm 1 Upgrade: Updating mon.b 2023-12-16T07:46:42.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:42 smithi007 bash[64479]: audit 2023-12-16T07:46:41.104982+0000 mon.a (mon.0) 99 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:42.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:42 smithi007 bash[64479]: cluster 2023-12-16T07:46:41.107229+0000 mgr.y (mgr.25477) 55 : cluster 0 pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:42.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:42 smithi007 bash[64479]: audit 2023-12-16T07:46:41.108945+0000 mon.a (mon.0) 100 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:46:42.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:42 smithi007 bash[64479]: audit 2023-12-16T07:46:41.110319+0000 mon.a (mon.0) 101 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:46:42.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:42 smithi007 bash[64479]: audit 2023-12-16T07:46:41.111791+0000 mon.a (mon.0) 102 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:46:42.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:42 smithi007 bash[64479]: cephadm 2023-12-16T07:46:41.113773+0000 mgr.y (mgr.25477) 56 : cephadm 1 Deploying daemon mon.b on smithi169 2023-12-16T07:46:42.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:42 smithi007 bash[65864]: cephadm 2023-12-16T07:46:41.097762+0000 mgr.y (mgr.25477) 54 : cephadm 1 Upgrade: Updating mon.b 2023-12-16T07:46:42.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:42 smithi007 bash[65864]: audit 2023-12-16T07:46:41.104982+0000 mon.a (mon.0) 99 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:42.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:42 smithi007 bash[65864]: cluster 2023-12-16T07:46:41.107229+0000 mgr.y (mgr.25477) 55 : cluster 0 pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:42.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:42 smithi007 bash[65864]: audit 2023-12-16T07:46:41.108945+0000 mon.a (mon.0) 100 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:46:42.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:42 smithi007 bash[65864]: audit 2023-12-16T07:46:41.110319+0000 mon.a (mon.0) 101 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:46:42.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:42 smithi007 bash[65864]: audit 2023-12-16T07:46:41.111791+0000 mon.a (mon.0) 102 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:46:42.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:42 smithi007 bash[65864]: cephadm 2023-12-16T07:46:41.113773+0000 mgr.y (mgr.25477) 56 : cephadm 1 Deploying daemon mon.b on smithi169 2023-12-16T07:46:43.617 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:43 smithi169 systemd[1]: Stopping Ceph mon.b for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:46:43.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:43 smithi169 bash[25083]: debug 2023-12-16T07:46:43.612+0000 7f4b421f0700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-16T07:46:43.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:43 smithi169 bash[25083]: debug 2023-12-16T07:46:43.612+0000 7f4b421f0700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-12-16T07:46:44.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:44 smithi169 bash[51495]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mon-b 2023-12-16T07:46:44.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:44 smithi169 bash[51567]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mon-b 2023-12-16T07:46:44.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:44 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.b.service: Succeeded. 2023-12-16T07:46:44.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:44 smithi169 systemd[1]: Stopped Ceph mon.b for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:46:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:44 smithi169 systemd[1]: Started Ceph mon.b for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:46:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.596+0000 7f6c5fa6cc80 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-16T07:46:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.596+0000 7f6c5fa6cc80 0 ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev), process ceph-mon, pid 7 2023-12-16T07:46:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.596+0000 7f6c5fa6cc80 0 pidfile_write: ignore empty --pid-file 2023-12-16T07:46:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.600+0000 7f6c5fa6cc80 0 load: jerasure load: lrc 2023-12-16T07:46:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: RocksDB version: 7.9.2 2023-12-16T07:46:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Git sha 0 2023-12-16T07:46:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Compile date 2023-12-15 16:32:04 2023-12-16T07:46:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: DB SUMMARY 2023-12-16T07:46:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: DB Session ID: 35ADZO99E11T6X04YC15 2023-12-16T07:46:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: CURRENT file: CURRENT 2023-12-16T07:46:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: IDENTITY file: IDENTITY 2023-12-16T07:46:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: MANIFEST file: MANIFEST-000009 size: 1263 Bytes 2023-12-16T07:46:45.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000030.sst 2023-12-16T07:46:45.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000028.log size: 5041372 ; 2023-12-16T07:46:45.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.error_if_exists: 0 2023-12-16T07:46:45.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.create_if_missing: 0 2023-12-16T07:46:45.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.paranoid_checks: 1 2023-12-16T07:46:45.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.flush_verify_memtable_count: 1 2023-12-16T07:46:45.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-12-16T07:46:45.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-12-16T07:46:45.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.env: 0x556e20183a60 2023-12-16T07:46:45.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.fs: PosixFileSystem 2023-12-16T07:46:45.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.info_log: 0x556e20d797a0 2023-12-16T07:46:45.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-16T07:46:45.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.statistics: (nil) 2023-12-16T07:46:45.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.use_fsync: 0 2023-12-16T07:46:45.936 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_log_file_size: 0 2023-12-16T07:46:45.936 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-16T07:46:45.936 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-16T07:46:45.936 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-16T07:46:45.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-16T07:46:45.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.allow_fallocate: 1 2023-12-16T07:46:45.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-16T07:46:45.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-16T07:46:45.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.use_direct_reads: 0 2023-12-16T07:46:45.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-16T07:46:45.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.create_missing_column_families: 0 2023-12-16T07:46:45.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.db_log_dir: 2023-12-16T07:46:45.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.wal_dir: 2023-12-16T07:46:45.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-16T07:46:45.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-16T07:46:45.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-16T07:46:45.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-16T07:46:45.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-16T07:46:45.938 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-16T07:46:45.938 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.advise_random_on_open: 1 2023-12-16T07:46:45.938 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-16T07:46:45.938 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.write_buffer_manager: 0x556e20d4d2c0 2023-12-16T07:46:45.938 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-16T07:46:45.938 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-16T07:46:45.938 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-16T07:46:45.938 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.rate_limiter: (nil) 2023-12-16T07:46:45.938 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-16T07:46:45.938 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-16T07:46:45.938 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-16T07:46:45.938 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-16T07:46:45.938 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.unordered_write: 0 2023-12-16T07:46:45.938 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-16T07:46:45.939 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-16T07:46:45.939 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-16T07:46:45.939 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-16T07:46:45.939 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.row_cache: None 2023-12-16T07:46:45.939 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.wal_filter: None 2023-12-16T07:46:45.939 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-16T07:46:45.939 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-16T07:46:45.939 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.two_write_queues: 0 2023-12-16T07:46:45.939 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.manual_wal_flush: 0 2023-12-16T07:46:45.939 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.wal_compression: 0 2023-12-16T07:46:45.939 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.atomic_flush: 0 2023-12-16T07:46:45.939 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-16T07:46:45.939 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-16T07:46:45.939 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-16T07:46:45.940 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.log_readahead_size: 0 2023-12-16T07:46:45.940 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.file_checksum_gen_factory: Unknown 2023-12-16T07:46:45.940 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.best_efforts_recovery: 0 2023-12-16T07:46:45.940 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-12-16T07:46:45.940 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-12-16T07:46:45.940 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.allow_data_in_errors: 0 2023-12-16T07:46:45.940 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.db_host_id: __hostname__ 2023-12-16T07:46:45.940 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.enforce_single_del_contracts: true 2023-12-16T07:46:45.940 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_background_jobs: 2 2023-12-16T07:46:45.940 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_background_compactions: -1 2023-12-16T07:46:45.940 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_subcompactions: 1 2023-12-16T07:46:45.940 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-16T07:46:45.940 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-16T07:46:45.940 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_total_wal_size: 0 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_open_files: -1 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.bytes_per_sync: 0 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_background_flushes: -1 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Compression algorithms supported: 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: kZSTD supported: 0 2023-12-16T07:46:45.941 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: kXpressCompression supported: 0 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: kZlibCompression supported: 1 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: kSnappyCompression supported: 1 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: kLZ4Compression supported: 1 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: kBZip2Compression supported: 0 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: DMutex implementation: pthread_mutex_t 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.merge_operator: 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compaction_filter: None 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compaction_filter_factory: None 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.sst_partitioner_factory: None 2023-12-16T07:46:45.942 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556e20d798c0) 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: cache_index_and_filter_blocks: 1 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: pin_top_level_index_and_filter: 1 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: index_type: 0 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: data_block_index_type: 0 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: index_shortening: 1 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: data_block_hash_table_util_ratio: 0.750000 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: checksum: 4 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: no_block_cache: 0 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: block_cache: 0x556e20dcd090 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: block_cache_name: BinnedLRUCache 2023-12-16T07:46:45.943 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: block_cache_options: 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: capacity : 536870912 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: num_shard_bits : 4 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: strict_capacity_limit : 0 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: high_pri_pool_ratio: 0.000 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: block_cache_compressed: (nil) 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: persistent_cache: (nil) 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: block_size: 4096 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: block_size_deviation: 10 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: block_restart_interval: 16 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: index_block_restart_interval: 1 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: metadata_block_size: 4096 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: partition_filters: 0 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: use_delta_encoding: 1 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: filter_policy: bloomfilter 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: whole_key_filtering: 1 2023-12-16T07:46:45.944 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: verify_compression: 0 2023-12-16T07:46:45.945 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: read_amp_bytes_per_bit: 0 2023-12-16T07:46:45.945 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: format_version: 5 2023-12-16T07:46:45.945 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: enable_index_compression: 1 2023-12-16T07:46:45.945 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: block_align: 0 2023-12-16T07:46:45.945 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: max_auto_readahead_size: 262144 2023-12-16T07:46:45.945 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: prepopulate_block_cache: 0 2023-12-16T07:46:45.945 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: initial_auto_readahead_size: 8192 2023-12-16T07:46:45.945 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: num_file_reads_for_auto_readahead: 2 2023-12-16T07:46:45.945 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-16T07:46:45.945 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-16T07:46:45.946 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compression: NoCompression 2023-12-16T07:46:45.946 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-16T07:46:45.946 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-16T07:46:45.946 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-16T07:46:45.946 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.num_levels: 7 2023-12-16T07:46:45.946 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compression_opts.level: 32767 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-16T07:46:45.947 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compression_opts.parallel_threads: 1 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compression_opts.enabled: false 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-16T07:46:45.948 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.arena_block_size: 1048576 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-16T07:46:45.949 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-16T07:46:45.950 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-16T07:46:45.950 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-16T07:46:45.950 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-16T07:46:45.950 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-16T07:46:45.950 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-16T07:46:45.950 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-16T07:46:45.950 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-16T07:46:45.950 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.table_properties_collectors: 2023-12-16T07:46:45.950 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.inplace_update_support: 0 2023-12-16T07:46:45.950 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-16T07:46:45.950 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-16T07:46:45.950 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-16T07:46:45.950 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.bloom_locality: 0 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.max_successive_merges: 0 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.force_consistency_checks: 1 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.ttl: 2592000 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.preclude_last_level_data_seconds: 0 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.preserve_internal_time_seconds: 0 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.enable_blob_files: false 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.min_blob_size: 0 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.blob_file_size: 268435456 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.blob_compression_type: NoCompression 2023-12-16T07:46:45.951 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.enable_blob_garbage_collection: false 2023-12-16T07:46:45.952 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-12-16T07:46:45.952 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-12-16T07:46:45.952 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.blob_compaction_readahead_size: 0 2023-12-16T07:46:45.952 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.blob_file_starting_level: 0 2023-12-16T07:46:45.952 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-12-16T07:46:45.952 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c52a2f700 3 rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2023-12-16T07:46:45.952 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-12-16T07:46:45.952 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 32, last_sequence is 18018, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-16T07:46:45.952 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2023-12-16T07:46:45.952 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b127e6f1-4aeb-4f74-a2b3-9a9e50d388de 2023-12-16T07:46:45.952 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702712805608841, "job": 1, "event": "recovery_started", "wal_files": [28]} 2023-12-16T07:46:45.952 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.604+0000 7f6c5fa6cc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2023-12-16T07:46:45.952 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.628+0000 7f6c5fa6cc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702712805630286, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3195391, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 18023, "largest_seqno": 19378, "table_properties": {"data_size": 3189641, "index_size": 3387, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1349, "raw_key_size": 12437, "raw_average_key_size": 23, "raw_value_size": 3178322, "raw_average_value_size": 5985, "num_data_blocks": 154, "num_entries": 531, "num_filter_entries": 531, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1702712805, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b127e6f1-4aeb-4f74-a2b3-9a9e50d388de", "db_session_id": "35ADZO99E11T6X04YC15", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2023-12-16T07:46:45.953 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.628+0000 7f6c5fa6cc80 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1702712805630586, "job": 1, "event": "recovery_finished"} 2023-12-16T07:46:45.953 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.628+0000 7f6c5fa6cc80 4 rocksdb: [db/version_set.cc:5047] Creating manifest 35 2023-12-16T07:46:45.953 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.628+0000 7f6c5fa6cc80 4 rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2023-12-16T07:46:45.953 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.632+0000 7f6c5fa6cc80 4 rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-12-16T07:46:45.953 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.632+0000 7f6c5fa6cc80 4 rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556e20e6c000 2023-12-16T07:46:45.953 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.632+0000 7f6c5fa6cc80 4 rocksdb: DB pointer 0x556e20e5a000 2023-12-16T07:46:45.953 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.632+0000 7f6c4e226700 4 rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-12-16T07:46:45.953 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.632+0000 7f6c4e226700 4 rocksdb: [db/db_impl/db_impl.cc:1111] 2023-12-16T07:46:45.953 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: ** DB Stats ** 2023-12-16T07:46:45.953 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:46:45.953 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-16T07:46:45.953 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-16T07:46:45.953 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T07:46:45.953 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-16T07:46:45.954 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-16T07:46:45.954 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-16T07:46:45.954 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: ** Compaction Stats [default] ** 2023-12-16T07:46:45.954 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-12-16T07:46:45.954 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-12-16T07:46:45.954 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: L0 1/0 3.05 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 513.6 0.01 0.00 1 0.006 0 0 0.0 0.0 2023-12-16T07:46:45.954 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: L6 1/0 9.93 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2023-12-16T07:46:45.954 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Sum 2/0 12.97 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 513.6 0.01 0.00 1 0.006 0 0 0.0 0.0 2023-12-16T07:46:45.954 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 513.6 0.01 0.00 1 0.006 0 0 0.0 0.0 2023-12-16T07:46:45.954 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: ** Compaction Stats [default] ** 2023-12-16T07:46:45.954 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-12-16T07:46:45.954 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-16T07:46:45.954 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 513.6 0.01 0.00 1 0.006 0 0 0.0 0.0 2023-12-16T07:46:45.954 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Flush(GB): cumulative 0.003, interval 0.003 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: AddFile(Total Files): cumulative 0, interval 0 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: AddFile(Keys): cumulative 0, interval 0 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Cumulative compaction: 0.00 GB write, 107.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Interval compaction: 0.00 GB write, 107.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Block cache BinnedLRUCache@0x556e20dcd090#7 capacity: 512.00 MB usage: 5.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: Block cache entry stats(count,size,portion): FilterBlock(1,1.41 KB,0.000268221%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: ** File Read Latency Histogram By Level [default] ** 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.632+0000 7f6c5fa6cc80 0 starting mon.b rank 2 at public addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] at bind addrs [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.632+0000 7f6c5fa6cc80 1 mon.b@-1(???) e3 preinit fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 2023-12-16T07:46:45.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.636+0000 7f6c5fa6cc80 0 mon.b@-1(???).mds e1 new map 2023-12-16T07:46:45.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.636+0000 7f6c5fa6cc80 0 mon.b@-1(???).mds e1 print_map 2023-12-16T07:46:45.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: e1 2023-12-16T07:46:45.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: enable_multiple, ever_enabled_multiple: 1,1 2023-12-16T07:46:45.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-16T07:46:45.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: legacy client fscid: -1 2023-12-16T07:46:45.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: 2023-12-16T07:46:45.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: No filesystems configured 2023-12-16T07:46:45.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.636+0000 7f6c5fa6cc80 0 mon.b@-1(???).osd e103 crush map has features 3314933000854323200, adjusting msgr requires 2023-12-16T07:46:45.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.636+0000 7f6c5fa6cc80 0 mon.b@-1(???).osd e103 crush map has features 432629239337189376, adjusting msgr requires 2023-12-16T07:46:45.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.636+0000 7f6c5fa6cc80 0 mon.b@-1(???).osd e103 crush map has features 432629239337189376, adjusting msgr requires 2023-12-16T07:46:45.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.636+0000 7f6c5fa6cc80 0 mon.b@-1(???).osd e103 crush map has features 432629239337189376, adjusting msgr requires 2023-12-16T07:46:45.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:45 smithi169 bash[51638]: debug 2023-12-16T07:46:45.636+0000 7f6c5fa6cc80 1 mon.b@-1(???).paxosservice(auth 1..25) refresh upgraded, format 0 -> 3 2023-12-16T07:46:48.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:46:48 smithi169 bash[49087]: ts=2023-12-16T07:46:48.662Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:46:50.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:46:50 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:46:50] "GET /metrics HTTP/1.1" 200 38164 "" "Prometheus/2.33.4" 2023-12-16T07:46:51.067 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:46:50 smithi007 bash[61260]: debug 2023-12-16T07:46:50.642+0000 7f2cea7a7700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-16T07:46:53.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:46.827367+0000 mon.c (mon.1) 3 : cluster 1 mon.c calling monitor election 2023-12-16T07:46:53.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:46.831398+0000 mon.b (mon.2) 2 : cluster 1 mon.b calling monitor election 2023-12-16T07:46:53.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.833821+0000 mon.c (mon.1) 4 : cluster 1 mon.c calling monitor election 2023-12-16T07:46:53.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.836245+0000 mon.a (mon.0) 115 : cluster 0 monmap e4: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T07:46:53.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.837333+0000 mon.a (mon.0) 116 : cluster 0 fsmap 2023-12-16T07:46:53.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.837402+0000 mon.a (mon.0) 117 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2023-12-16T07:46:53.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.837974+0000 mon.a (mon.0) 118 : cluster 0 mgrmap e40: y(active, since 58s), standbys: x 2023-12-16T07:46:53.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.838368+0000 mon.a (mon.0) 119 : cluster 3 Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-12-16T07:46:53.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.844056+0000 mon.a (mon.0) 120 : cluster 1 overall HEALTH_OK 2023-12-16T07:46:53.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.844660+0000 mon.a (mon.0) 121 : cluster 1 mon.a calling monitor election 2023-12-16T07:46:53.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.847688+0000 mon.a (mon.0) 122 : cluster 1 mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-12-16T07:46:53.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.856208+0000 mon.a (mon.0) 123 : cluster 0 monmap e4: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T07:46:53.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.856991+0000 mon.a (mon.0) 124 : cluster 0 fsmap 2023-12-16T07:46:53.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.857045+0000 mon.a (mon.0) 125 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2023-12-16T07:46:53.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.857540+0000 mon.a (mon.0) 126 : cluster 0 mgrmap e40: y(active, since 58s), standbys: x 2023-12-16T07:46:53.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.857777+0000 mon.a (mon.0) 127 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-12-16T07:46:53.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.857816+0000 mon.a (mon.0) 128 : cluster 1 Cluster is now healthy 2023-12-16T07:46:53.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: cluster 2023-12-16T07:46:51.866151+0000 mon.a (mon.0) 129 : cluster 1 overall HEALTH_OK 2023-12-16T07:46:53.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: audit 2023-12-16T07:46:51.870815+0000 mon.a (mon.0) 130 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:53.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:52 smithi169 bash[51638]: audit 2023-12-16T07:46:51.878748+0000 mon.a (mon.0) 131 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:46.827367+0000 mon.c (mon.1) 3 : cluster 1 mon.c calling monitor election 2023-12-16T07:46:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:46.831398+0000 mon.b (mon.2) 2 : cluster 1 mon.b calling monitor election 2023-12-16T07:46:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.833821+0000 mon.c (mon.1) 4 : cluster 1 mon.c calling monitor election 2023-12-16T07:46:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.836245+0000 mon.a (mon.0) 115 : cluster 0 monmap e4: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T07:46:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.837333+0000 mon.a (mon.0) 116 : cluster 0 fsmap 2023-12-16T07:46:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.837402+0000 mon.a (mon.0) 117 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2023-12-16T07:46:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.837974+0000 mon.a (mon.0) 118 : cluster 0 mgrmap e40: y(active, since 58s), standbys: x 2023-12-16T07:46:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.838368+0000 mon.a (mon.0) 119 : cluster 3 Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-12-16T07:46:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.844056+0000 mon.a (mon.0) 120 : cluster 1 overall HEALTH_OK 2023-12-16T07:46:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.844660+0000 mon.a (mon.0) 121 : cluster 1 mon.a calling monitor election 2023-12-16T07:46:53.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.847688+0000 mon.a (mon.0) 122 : cluster 1 mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-12-16T07:46:53.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.856208+0000 mon.a (mon.0) 123 : cluster 0 monmap e4: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T07:46:53.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.856991+0000 mon.a (mon.0) 124 : cluster 0 fsmap 2023-12-16T07:46:53.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.857045+0000 mon.a (mon.0) 125 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2023-12-16T07:46:53.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.857540+0000 mon.a (mon.0) 126 : cluster 0 mgrmap e40: y(active, since 58s), standbys: x 2023-12-16T07:46:53.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.857777+0000 mon.a (mon.0) 127 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-12-16T07:46:53.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.857816+0000 mon.a (mon.0) 128 : cluster 1 Cluster is now healthy 2023-12-16T07:46:53.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: cluster 2023-12-16T07:46:51.866151+0000 mon.a (mon.0) 129 : cluster 1 overall HEALTH_OK 2023-12-16T07:46:53.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: audit 2023-12-16T07:46:51.870815+0000 mon.a (mon.0) 130 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:53.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[64479]: audit 2023-12-16T07:46:51.878748+0000 mon.a (mon.0) 131 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:53.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:46.827367+0000 mon.c (mon.1) 3 : cluster 1 mon.c calling monitor election 2023-12-16T07:46:53.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:46.831398+0000 mon.b (mon.2) 2 : cluster 1 mon.b calling monitor election 2023-12-16T07:46:53.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.833821+0000 mon.c (mon.1) 4 : cluster 1 mon.c calling monitor election 2023-12-16T07:46:53.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.836245+0000 mon.a (mon.0) 115 : cluster 0 monmap e4: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T07:46:53.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.837333+0000 mon.a (mon.0) 116 : cluster 0 fsmap 2023-12-16T07:46:53.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.837402+0000 mon.a (mon.0) 117 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2023-12-16T07:46:53.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.837974+0000 mon.a (mon.0) 118 : cluster 0 mgrmap e40: y(active, since 58s), standbys: x 2023-12-16T07:46:53.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.838368+0000 mon.a (mon.0) 119 : cluster 3 Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-12-16T07:46:53.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.844056+0000 mon.a (mon.0) 120 : cluster 1 overall HEALTH_OK 2023-12-16T07:46:53.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.844660+0000 mon.a (mon.0) 121 : cluster 1 mon.a calling monitor election 2023-12-16T07:46:53.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.847688+0000 mon.a (mon.0) 122 : cluster 1 mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-12-16T07:46:53.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.856208+0000 mon.a (mon.0) 123 : cluster 0 monmap e4: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-12-16T07:46:53.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.856991+0000 mon.a (mon.0) 124 : cluster 0 fsmap 2023-12-16T07:46:53.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.857045+0000 mon.a (mon.0) 125 : cluster 0 osdmap e103: 8 total, 8 up, 8 in 2023-12-16T07:46:53.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.857540+0000 mon.a (mon.0) 126 : cluster 0 mgrmap e40: y(active, since 58s), standbys: x 2023-12-16T07:46:53.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.857777+0000 mon.a (mon.0) 127 : cluster 1 Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-12-16T07:46:53.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.857816+0000 mon.a (mon.0) 128 : cluster 1 Cluster is now healthy 2023-12-16T07:46:53.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: cluster 2023-12-16T07:46:51.866151+0000 mon.a (mon.0) 129 : cluster 1 overall HEALTH_OK 2023-12-16T07:46:53.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: audit 2023-12-16T07:46:51.870815+0000 mon.a (mon.0) 130 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:53.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:52 smithi007 bash[65864]: audit 2023-12-16T07:46:51.878748+0000 mon.a (mon.0) 131 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:54.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:53 smithi007 bash[64479]: cluster 2023-12-16T07:46:53.114048+0000 mgr.y (mgr.25477) 63 : cluster 0 pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:54.140 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:53 smithi007 bash[64479]: audit 2023-12-16T07:46:53.199936+0000 mon.a (mon.0) 132 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:46:54.140 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:53 smithi007 bash[65864]: cluster 2023-12-16T07:46:53.114048+0000 mgr.y (mgr.25477) 63 : cluster 0 pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:54.141 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:53 smithi007 bash[65864]: audit 2023-12-16T07:46:53.199936+0000 mon.a (mon.0) 132 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:46:54.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:53 smithi169 bash[51638]: cluster 2023-12-16T07:46:53.114048+0000 mgr.y (mgr.25477) 63 : cluster 0 pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:54.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:53 smithi169 bash[51638]: audit 2023-12-16T07:46:53.199936+0000 mon.a (mon.0) 132 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:46:55.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:54 smithi169 bash[51638]: audit 2023-12-16T07:46:54.137610+0000 mgr.y (mgr.25477) 64 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:55.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:54 smithi007 bash[65864]: audit 2023-12-16T07:46:54.137610+0000 mgr.y (mgr.25477) 64 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:55.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:54 smithi007 bash[64479]: audit 2023-12-16T07:46:54.137610+0000 mgr.y (mgr.25477) 64 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:46:56.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:55 smithi169 bash[51638]: cluster 2023-12-16T07:46:55.115926+0000 mgr.y (mgr.25477) 65 : cluster 0 pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:56.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:55 smithi007 bash[64479]: cluster 2023-12-16T07:46:55.115926+0000 mgr.y (mgr.25477) 65 : cluster 0 pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:56.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:55 smithi007 bash[65864]: cluster 2023-12-16T07:46:55.115926+0000 mgr.y (mgr.25477) 65 : cluster 0 pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:46:58.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:58 smithi169 bash[51638]: cluster 2023-12-16T07:46:57.116648+0000 mgr.y (mgr.25477) 66 : cluster 0 pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:58.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:58 smithi007 bash[64479]: cluster 2023-12-16T07:46:57.116648+0000 mgr.y (mgr.25477) 66 : cluster 0 pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:58.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:58 smithi007 bash[65864]: cluster 2023-12-16T07:46:57.116648+0000 mgr.y (mgr.25477) 66 : cluster 0 pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:46:58.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:46:58 smithi169 bash[49087]: ts=2023-12-16T07:46:58.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:46:59.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:59 smithi007 bash[64479]: audit 2023-12-16T07:46:58.312027+0000 mon.a (mon.0) 133 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:59.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:46:59 smithi007 bash[64479]: audit 2023-12-16T07:46:58.323780+0000 mon.a (mon.0) 134 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:59.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:59 smithi007 bash[65864]: audit 2023-12-16T07:46:58.312027+0000 mon.a (mon.0) 133 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:59.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:46:59 smithi007 bash[65864]: audit 2023-12-16T07:46:58.323780+0000 mon.a (mon.0) 134 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:59 smithi169 bash[51638]: audit 2023-12-16T07:46:58.312027+0000 mon.a (mon.0) 133 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:46:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:46:59 smithi169 bash[51638]: audit 2023-12-16T07:46:58.323780+0000 mon.a (mon.0) 134 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:00.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:00 smithi007 bash[65864]: cluster 2023-12-16T07:46:59.118187+0000 mgr.y (mgr.25477) 67 : cluster 0 pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:00.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:00 smithi007 bash[65864]: audit 2023-12-16T07:46:59.321196+0000 mon.a (mon.0) 135 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:00.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:00 smithi007 bash[65864]: audit 2023-12-16T07:46:59.329898+0000 mon.a (mon.0) 136 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:00.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:47:00 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:47:00] "GET /metrics HTTP/1.1" 200 38173 "" "Prometheus/2.33.4" 2023-12-16T07:47:00.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:00 smithi007 bash[64479]: cluster 2023-12-16T07:46:59.118187+0000 mgr.y (mgr.25477) 67 : cluster 0 pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:00.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:00 smithi007 bash[64479]: audit 2023-12-16T07:46:59.321196+0000 mon.a (mon.0) 135 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:00.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:00 smithi007 bash[64479]: audit 2023-12-16T07:46:59.329898+0000 mon.a (mon.0) 136 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:00.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:00 smithi169 bash[51638]: cluster 2023-12-16T07:46:59.118187+0000 mgr.y (mgr.25477) 67 : cluster 0 pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:00.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:00 smithi169 bash[51638]: audit 2023-12-16T07:46:59.321196+0000 mon.a (mon.0) 135 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:00.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:00 smithi169 bash[51638]: audit 2023-12-16T07:46:59.329898+0000 mon.a (mon.0) 136 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:02.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:02 smithi169 bash[51638]: cluster 2023-12-16T07:47:01.118957+0000 mgr.y (mgr.25477) 68 : cluster 0 pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:02.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:02 smithi007 bash[65864]: cluster 2023-12-16T07:47:01.118957+0000 mgr.y (mgr.25477) 68 : cluster 0 pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:02.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:02 smithi007 bash[64479]: cluster 2023-12-16T07:47:01.118957+0000 mgr.y (mgr.25477) 68 : cluster 0 pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:03.165 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:47:03.937 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:47:03.937 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (3m) 36s ago 10m 16.4M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:47:03.937 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (2m) 5s ago 10m 43.6M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:47:03.937 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (117s) 36s ago 8m 45.7M - 3.5 e1d6a67b021e 8d8009d0c83f 2023-12-16T07:47:03.937 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283,8765 running (106s) 5s ago 16m 433M - 19.0.0-191-g6d933094 a29c258b5d6f 5295ba35c2cd 2023-12-16T07:47:03.937 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (2m) 36s ago 18m 478M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:47:03.937 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (81s) 36s ago 18m 39.0M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 5a266b45f3ce 2023-12-16T07:47:03.937 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (19s) 5s ago 16m 23.6M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1fc9bf864dda 2023-12-16T07:47:03.938 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (45s) 36s ago 17m 22.5M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1dc91c31fc27 2023-12-16T07:47:03.938 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (3m) 36s ago 10m 11.0M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:47:03.938 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (3m) 5s ago 10m 11.3M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:47:03.938 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (16m) 36s ago 16m 58.3M 2782M 17.2.0 e1d6a67b021e dbc4d9538238 2023-12-16T07:47:03.938 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (15m) 36s ago 15m 59.2M 2782M 17.2.0 e1d6a67b021e 14637e0f389c 2023-12-16T07:47:03.938 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (14m) 36s ago 14m 57.9M 2782M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:47:03.938 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (14m) 36s ago 14m 57.1M 2782M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:47:03.938 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (13m) 5s ago 13m 58.5M 3037M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:47:03.938 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (13m) 5s ago 13m 55.3M 3037M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:47:03.938 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (12m) 5s ago 12m 55.8M 3037M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:47:03.938 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (11m) 5s ago 11m 56.9M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:47:03.938 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (112s) 5s ago 10m 59.0M - 2.33.4 514e6a882f6e 2652ce1b8540 2023-12-16T07:47:03.938 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (8m) 36s ago 8m 87.0M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:47:03.939 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (8m) 5s ago 8m 87.4M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:47:03.939 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (8m) 36s ago 8m 86.8M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:47:03.939 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (8m) 5s ago 8m 87.0M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:47:04.391 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:47:04.391 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:47:04.391 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 3 2023-12-16T07:47:04.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:47:04.391 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:47:04.391 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:47:04.391 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:47:04.391 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:47:04.392 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2023-12-16T07:47:04.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:47:04.392 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:47:04.392 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:47:04.392 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:47:04.392 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:47:04.392 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2023-12-16T07:47:04.392 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 5 2023-12-16T07:47:04.392 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:47:04.392 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:47:04.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:04 smithi007 bash[64479]: cluster 2023-12-16T07:47:03.119622+0000 mgr.y (mgr.25477) 69 : cluster 0 pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:04.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:04 smithi007 bash[64479]: audit 2023-12-16T07:47:03.146264+0000 mgr.y (mgr.25477) 70 : audit 0 from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:04.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:04 smithi007 bash[65864]: cluster 2023-12-16T07:47:03.119622+0000 mgr.y (mgr.25477) 69 : cluster 0 pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:04.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:04 smithi007 bash[65864]: audit 2023-12-16T07:47:03.146264+0000 mgr.y (mgr.25477) 70 : audit 0 from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:04.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:04 smithi169 bash[51638]: cluster 2023-12-16T07:47:03.119622+0000 mgr.y (mgr.25477) 69 : cluster 0 pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:04.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:04 smithi169 bash[51638]: audit 2023-12-16T07:47:03.146264+0000 mgr.y (mgr.25477) 70 : audit 0 from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:04.781 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:47:04.781 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:47:04.781 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:47:04.781 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:47:04.781 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2023-12-16T07:47:04.781 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr", 2023-12-16T07:47:04.781 INFO:teuthology.orchestra.run.smithi007.stdout: "mon" 2023-12-16T07:47:04.781 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:47:04.781 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "5/23 daemons upgraded", 2023-12-16T07:47:04.781 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading mon daemons", 2023-12-16T07:47:04.781 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:47:04.782 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:47:05.246 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:47:05.674 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:05 smithi169 bash[51638]: audit 2023-12-16T07:47:03.543378+0000 mgr.y (mgr.25477) 71 : audit 0 from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:05.674 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:05 smithi169 bash[51638]: audit 2023-12-16T07:47:03.918106+0000 mgr.y (mgr.25477) 72 : audit 0 from='client.34215 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:05.674 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:05 smithi169 bash[51638]: audit 2023-12-16T07:47:04.147158+0000 mgr.y (mgr.25477) 73 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:05.674 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:05 smithi169 bash[51638]: audit 2023-12-16T07:47:04.388285+0000 mon.a (mon.0) 137 : audit 0 from='client.? 172.21.15.7:0/1199963818' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:05.674 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:05 smithi169 bash[51638]: audit 2023-12-16T07:47:05.241183+0000 mon.c (mon.1) 5 : audit 0 from='client.? 172.21.15.7:0/76258309' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:47:05.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:05 smithi007 bash[65864]: audit 2023-12-16T07:47:03.543378+0000 mgr.y (mgr.25477) 71 : audit 0 from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:05.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:05 smithi007 bash[65864]: audit 2023-12-16T07:47:03.918106+0000 mgr.y (mgr.25477) 72 : audit 0 from='client.34215 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:05.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:05 smithi007 bash[65864]: audit 2023-12-16T07:47:04.147158+0000 mgr.y (mgr.25477) 73 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:05.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:05 smithi007 bash[65864]: audit 2023-12-16T07:47:04.388285+0000 mon.a (mon.0) 137 : audit 0 from='client.? 172.21.15.7:0/1199963818' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:05.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:05 smithi007 bash[65864]: audit 2023-12-16T07:47:05.241183+0000 mon.c (mon.1) 5 : audit 0 from='client.? 172.21.15.7:0/76258309' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:47:05.823 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:05 smithi007 bash[64479]: audit 2023-12-16T07:47:03.543378+0000 mgr.y (mgr.25477) 71 : audit 0 from='client.44143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:05.823 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:05 smithi007 bash[64479]: audit 2023-12-16T07:47:03.918106+0000 mgr.y (mgr.25477) 72 : audit 0 from='client.34215 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:05.823 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:05 smithi007 bash[64479]: audit 2023-12-16T07:47:04.147158+0000 mgr.y (mgr.25477) 73 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:05.823 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:05 smithi007 bash[64479]: audit 2023-12-16T07:47:04.388285+0000 mon.a (mon.0) 137 : audit 0 from='client.? 172.21.15.7:0/1199963818' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:05.823 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:05 smithi007 bash[64479]: audit 2023-12-16T07:47:05.241183+0000 mon.c (mon.1) 5 : audit 0 from='client.? 172.21.15.7:0/76258309' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:47:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:06 smithi169 bash[51638]: audit 2023-12-16T07:47:04.778221+0000 mgr.y (mgr.25477) 74 : audit 0 from='client.34227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:06 smithi169 bash[51638]: cluster 2023-12-16T07:47:05.121217+0000 mgr.y (mgr.25477) 75 : cluster 0 pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:06.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:06 smithi007 bash[64479]: audit 2023-12-16T07:47:04.778221+0000 mgr.y (mgr.25477) 74 : audit 0 from='client.34227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:06.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:06 smithi007 bash[64479]: cluster 2023-12-16T07:47:05.121217+0000 mgr.y (mgr.25477) 75 : cluster 0 pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:06.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:06 smithi007 bash[65864]: audit 2023-12-16T07:47:04.778221+0000 mgr.y (mgr.25477) 74 : audit 0 from='client.34227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:06.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:06 smithi007 bash[65864]: cluster 2023-12-16T07:47:05.121217+0000 mgr.y (mgr.25477) 75 : cluster 0 pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:08.662 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:08 smithi169 bash[51638]: cluster 2023-12-16T07:47:07.121962+0000 mgr.y (mgr.25477) 76 : cluster 0 pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:08.662 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:08 smithi169 bash[51638]: audit 2023-12-16T07:47:08.200757+0000 mon.a (mon.0) 138 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:47:08.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:08 smithi007 bash[64479]: cluster 2023-12-16T07:47:07.121962+0000 mgr.y (mgr.25477) 76 : cluster 0 pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:08.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:08 smithi007 bash[64479]: audit 2023-12-16T07:47:08.200757+0000 mon.a (mon.0) 138 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:47:08.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:08 smithi007 bash[65864]: cluster 2023-12-16T07:47:07.121962+0000 mgr.y (mgr.25477) 76 : cluster 0 pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:08.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:08 smithi007 bash[65864]: audit 2023-12-16T07:47:08.200757+0000 mon.a (mon.0) 138 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:47:08.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:47:08 smithi169 bash[49087]: ts=2023-12-16T07:47:08.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:47:10.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[65864]: audit 2023-12-16T07:47:08.983038+0000 mon.a (mon.0) 139 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:10.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[65864]: audit 2023-12-16T07:47:08.994525+0000 mon.a (mon.0) 140 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:10.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[65864]: audit 2023-12-16T07:47:08.997304+0000 mon.a (mon.0) 141 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:10.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[65864]: audit 2023-12-16T07:47:08.999046+0000 mon.a (mon.0) 142 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:47:10.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[65864]: audit 2023-12-16T07:47:09.008302+0000 mon.a (mon.0) 143 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:10.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[65864]: cephadm 2023-12-16T07:47:09.083147+0000 mgr.y (mgr.25477) 77 : cephadm 1 Reconfiguring mon.a (monmap changed)... 2023-12-16T07:47:10.252 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[65864]: audit 2023-12-16T07:47:09.083968+0000 mon.a (mon.0) 144 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:47:10.252 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[65864]: audit 2023-12-16T07:47:09.086133+0000 mon.a (mon.0) 145 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:47:10.252 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[65864]: audit 2023-12-16T07:47:09.087856+0000 mon.a (mon.0) 146 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:10.252 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[65864]: cephadm 2023-12-16T07:47:09.089695+0000 mgr.y (mgr.25477) 78 : cephadm 1 Reconfiguring daemon mon.a on smithi007 2023-12-16T07:47:10.252 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[65864]: cluster 2023-12-16T07:47:09.123480+0000 mgr.y (mgr.25477) 79 : cluster 0 pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:10.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[64479]: audit 2023-12-16T07:47:08.983038+0000 mon.a (mon.0) 139 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:10.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[64479]: audit 2023-12-16T07:47:08.994525+0000 mon.a (mon.0) 140 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:10.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[64479]: audit 2023-12-16T07:47:08.997304+0000 mon.a (mon.0) 141 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:10.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[64479]: audit 2023-12-16T07:47:08.999046+0000 mon.a (mon.0) 142 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:47:10.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[64479]: audit 2023-12-16T07:47:09.008302+0000 mon.a (mon.0) 143 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:10.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[64479]: cephadm 2023-12-16T07:47:09.083147+0000 mgr.y (mgr.25477) 77 : cephadm 1 Reconfiguring mon.a (monmap changed)... 2023-12-16T07:47:10.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[64479]: audit 2023-12-16T07:47:09.083968+0000 mon.a (mon.0) 144 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:47:10.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[64479]: audit 2023-12-16T07:47:09.086133+0000 mon.a (mon.0) 145 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:47:10.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[64479]: audit 2023-12-16T07:47:09.087856+0000 mon.a (mon.0) 146 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:10.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[64479]: cephadm 2023-12-16T07:47:09.089695+0000 mgr.y (mgr.25477) 78 : cephadm 1 Reconfiguring daemon mon.a on smithi007 2023-12-16T07:47:10.253 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:09 smithi007 bash[64479]: cluster 2023-12-16T07:47:09.123480+0000 mgr.y (mgr.25477) 79 : cluster 0 pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:10.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:09 smithi169 bash[51638]: audit 2023-12-16T07:47:08.983038+0000 mon.a (mon.0) 139 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:10.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:09 smithi169 bash[51638]: audit 2023-12-16T07:47:08.994525+0000 mon.a (mon.0) 140 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:10.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:09 smithi169 bash[51638]: audit 2023-12-16T07:47:08.997304+0000 mon.a (mon.0) 141 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:10.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:09 smithi169 bash[51638]: audit 2023-12-16T07:47:08.999046+0000 mon.a (mon.0) 142 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:47:10.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:09 smithi169 bash[51638]: audit 2023-12-16T07:47:09.008302+0000 mon.a (mon.0) 143 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:10.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:09 smithi169 bash[51638]: cephadm 2023-12-16T07:47:09.083147+0000 mgr.y (mgr.25477) 77 : cephadm 1 Reconfiguring mon.a (monmap changed)... 2023-12-16T07:47:10.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:09 smithi169 bash[51638]: audit 2023-12-16T07:47:09.083968+0000 mon.a (mon.0) 144 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:47:10.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:09 smithi169 bash[51638]: audit 2023-12-16T07:47:09.086133+0000 mon.a (mon.0) 145 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:47:10.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:09 smithi169 bash[51638]: audit 2023-12-16T07:47:09.087856+0000 mon.a (mon.0) 146 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:10.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:09 smithi169 bash[51638]: cephadm 2023-12-16T07:47:09.089695+0000 mgr.y (mgr.25477) 78 : cephadm 1 Reconfiguring daemon mon.a on smithi007 2023-12-16T07:47:10.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:09 smithi169 bash[51638]: cluster 2023-12-16T07:47:09.123480+0000 mgr.y (mgr.25477) 79 : cluster 0 pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:10.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:47:10 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:47:10] "GET /metrics HTTP/1.1" 200 38169 "" "Prometheus/2.33.4" 2023-12-16T07:47:12.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:12 smithi007 bash[64479]: cluster 2023-12-16T07:47:11.124216+0000 mgr.y (mgr.25477) 80 : cluster 0 pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:12.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:12 smithi007 bash[64479]: audit 2023-12-16T07:47:11.750172+0000 mon.a (mon.0) 147 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:12.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:12 smithi007 bash[64479]: audit 2023-12-16T07:47:11.758740+0000 mon.a (mon.0) 148 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:12 smithi007 bash[64479]: audit 2023-12-16T07:47:11.811628+0000 mon.a (mon.0) 149 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:47:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:12 smithi007 bash[64479]: audit 2023-12-16T07:47:11.813694+0000 mon.a (mon.0) 150 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:47:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:12 smithi007 bash[64479]: audit 2023-12-16T07:47:11.815832+0000 mon.a (mon.0) 151 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:12.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:12 smithi007 bash[65864]: cluster 2023-12-16T07:47:11.124216+0000 mgr.y (mgr.25477) 80 : cluster 0 pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:12.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:12 smithi007 bash[65864]: audit 2023-12-16T07:47:11.750172+0000 mon.a (mon.0) 147 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:12.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:12 smithi007 bash[65864]: audit 2023-12-16T07:47:11.758740+0000 mon.a (mon.0) 148 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:12.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:12 smithi007 bash[65864]: audit 2023-12-16T07:47:11.811628+0000 mon.a (mon.0) 149 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:47:12.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:12 smithi007 bash[65864]: audit 2023-12-16T07:47:11.813694+0000 mon.a (mon.0) 150 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:47:12.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:12 smithi007 bash[65864]: audit 2023-12-16T07:47:11.815832+0000 mon.a (mon.0) 151 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:12 smithi169 bash[51638]: cluster 2023-12-16T07:47:11.124216+0000 mgr.y (mgr.25477) 80 : cluster 0 pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:12 smithi169 bash[51638]: audit 2023-12-16T07:47:11.750172+0000 mon.a (mon.0) 147 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:12 smithi169 bash[51638]: audit 2023-12-16T07:47:11.758740+0000 mon.a (mon.0) 148 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:12 smithi169 bash[51638]: audit 2023-12-16T07:47:11.811628+0000 mon.a (mon.0) 149 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:47:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:12 smithi169 bash[51638]: audit 2023-12-16T07:47:11.813694+0000 mon.a (mon.0) 150 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:47:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:12 smithi169 bash[51638]: audit 2023-12-16T07:47:11.815832+0000 mon.a (mon.0) 151 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:13.478 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:13 smithi007 bash[64479]: cephadm 2023-12-16T07:47:11.810826+0000 mgr.y (mgr.25477) 81 : cephadm 1 Reconfiguring mgr.y (monmap changed)... 2023-12-16T07:47:13.478 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:13 smithi007 bash[64479]: cephadm 2023-12-16T07:47:11.817973+0000 mgr.y (mgr.25477) 82 : cephadm 1 Reconfiguring daemon mgr.y on smithi007 2023-12-16T07:47:13.478 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:13 smithi007 bash[65864]: cephadm 2023-12-16T07:47:11.810826+0000 mgr.y (mgr.25477) 81 : cephadm 1 Reconfiguring mgr.y (monmap changed)... 2023-12-16T07:47:13.478 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:13 smithi007 bash[65864]: cephadm 2023-12-16T07:47:11.817973+0000 mgr.y (mgr.25477) 82 : cephadm 1 Reconfiguring daemon mgr.y on smithi007 2023-12-16T07:47:13.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:13 smithi169 bash[51638]: cephadm 2023-12-16T07:47:11.810826+0000 mgr.y (mgr.25477) 81 : cephadm 1 Reconfiguring mgr.y (monmap changed)... 2023-12-16T07:47:13.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:13 smithi169 bash[51638]: cephadm 2023-12-16T07:47:11.817973+0000 mgr.y (mgr.25477) 82 : cephadm 1 Reconfiguring daemon mgr.y on smithi007 2023-12-16T07:47:14.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:14 smithi007 bash[65864]: cluster 2023-12-16T07:47:13.124945+0000 mgr.y (mgr.25477) 83 : cluster 0 pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:14.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:14 smithi007 bash[64479]: cluster 2023-12-16T07:47:13.124945+0000 mgr.y (mgr.25477) 83 : cluster 0 pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:14.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:14 smithi169 bash[51638]: cluster 2023-12-16T07:47:13.124945+0000 mgr.y (mgr.25477) 83 : cluster 0 pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:15 smithi169 bash[51638]: audit 2023-12-16T07:47:14.155989+0000 mgr.y (mgr.25477) 84 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:15 smithi169 bash[51638]: audit 2023-12-16T07:47:14.343721+0000 mon.a (mon.0) 152 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:15 smithi169 bash[51638]: audit 2023-12-16T07:47:14.354141+0000 mon.a (mon.0) 153 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:15 smithi169 bash[51638]: cephadm 2023-12-16T07:47:14.390877+0000 mgr.y (mgr.25477) 85 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2023-12-16T07:47:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:15 smithi169 bash[51638]: audit 2023-12-16T07:47:14.391512+0000 mon.a (mon.0) 154 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:47:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:15 smithi169 bash[51638]: audit 2023-12-16T07:47:14.393332+0000 mon.a (mon.0) 155 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:47:15.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:15 smithi169 bash[51638]: audit 2023-12-16T07:47:14.395134+0000 mon.a (mon.0) 156 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:15.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:15 smithi169 bash[51638]: cephadm 2023-12-16T07:47:14.397214+0000 mgr.y (mgr.25477) 86 : cephadm 1 Reconfiguring daemon mon.c on smithi007 2023-12-16T07:47:15.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[64479]: audit 2023-12-16T07:47:14.155989+0000 mgr.y (mgr.25477) 84 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:15.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[64479]: audit 2023-12-16T07:47:14.343721+0000 mon.a (mon.0) 152 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:15.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[64479]: audit 2023-12-16T07:47:14.354141+0000 mon.a (mon.0) 153 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:15.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[64479]: cephadm 2023-12-16T07:47:14.390877+0000 mgr.y (mgr.25477) 85 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2023-12-16T07:47:15.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[64479]: audit 2023-12-16T07:47:14.391512+0000 mon.a (mon.0) 154 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:47:15.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[64479]: audit 2023-12-16T07:47:14.393332+0000 mon.a (mon.0) 155 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:47:15.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[64479]: audit 2023-12-16T07:47:14.395134+0000 mon.a (mon.0) 156 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:15.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[64479]: cephadm 2023-12-16T07:47:14.397214+0000 mgr.y (mgr.25477) 86 : cephadm 1 Reconfiguring daemon mon.c on smithi007 2023-12-16T07:47:15.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[65864]: audit 2023-12-16T07:47:14.155989+0000 mgr.y (mgr.25477) 84 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:15.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[65864]: audit 2023-12-16T07:47:14.343721+0000 mon.a (mon.0) 152 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:15.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[65864]: audit 2023-12-16T07:47:14.354141+0000 mon.a (mon.0) 153 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:15.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[65864]: cephadm 2023-12-16T07:47:14.390877+0000 mgr.y (mgr.25477) 85 : cephadm 1 Reconfiguring mon.c (monmap changed)... 2023-12-16T07:47:15.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[65864]: audit 2023-12-16T07:47:14.391512+0000 mon.a (mon.0) 154 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:47:15.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[65864]: audit 2023-12-16T07:47:14.393332+0000 mon.a (mon.0) 155 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:47:15.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[65864]: audit 2023-12-16T07:47:14.395134+0000 mon.a (mon.0) 156 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:15.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:15 smithi007 bash[65864]: cephadm 2023-12-16T07:47:14.397214+0000 mgr.y (mgr.25477) 86 : cephadm 1 Reconfiguring daemon mon.c on smithi007 2023-12-16T07:47:16.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:16 smithi007 bash[64479]: cluster 2023-12-16T07:47:15.126682+0000 mgr.y (mgr.25477) 87 : cluster 0 pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:16.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:16 smithi007 bash[65864]: cluster 2023-12-16T07:47:15.126682+0000 mgr.y (mgr.25477) 87 : cluster 0 pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:16.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:16 smithi169 bash[51638]: cluster 2023-12-16T07:47:15.126682+0000 mgr.y (mgr.25477) 87 : cluster 0 pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:18.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:17 smithi169 bash[51638]: audit 2023-12-16T07:47:16.889936+0000 mon.a (mon.0) 157 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:18.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:17 smithi169 bash[51638]: audit 2023-12-16T07:47:16.899038+0000 mon.a (mon.0) 158 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:18.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:17 smithi169 bash[51638]: cephadm 2023-12-16T07:47:16.918457+0000 mgr.y (mgr.25477) 88 : cephadm 1 Reconfiguring osd.0 (monmap changed)... 2023-12-16T07:47:18.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:17 smithi169 bash[51638]: audit 2023-12-16T07:47:16.919043+0000 mon.a (mon.0) 159 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-16T07:47:18.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:17 smithi169 bash[51638]: audit 2023-12-16T07:47:16.920660+0000 mon.a (mon.0) 160 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:18.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:17 smithi169 bash[51638]: cephadm 2023-12-16T07:47:16.925682+0000 mgr.y (mgr.25477) 89 : cephadm 1 Reconfiguring daemon osd.0 on smithi007 2023-12-16T07:47:18.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:17 smithi169 bash[51638]: cluster 2023-12-16T07:47:17.127374+0000 mgr.y (mgr.25477) 90 : cluster 0 pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:18.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:17 smithi007 bash[65864]: audit 2023-12-16T07:47:16.889936+0000 mon.a (mon.0) 157 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:18.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:17 smithi007 bash[65864]: audit 2023-12-16T07:47:16.899038+0000 mon.a (mon.0) 158 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:18.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:17 smithi007 bash[65864]: cephadm 2023-12-16T07:47:16.918457+0000 mgr.y (mgr.25477) 88 : cephadm 1 Reconfiguring osd.0 (monmap changed)... 2023-12-16T07:47:18.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:17 smithi007 bash[65864]: audit 2023-12-16T07:47:16.919043+0000 mon.a (mon.0) 159 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-16T07:47:18.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:17 smithi007 bash[65864]: audit 2023-12-16T07:47:16.920660+0000 mon.a (mon.0) 160 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:18.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:17 smithi007 bash[65864]: cephadm 2023-12-16T07:47:16.925682+0000 mgr.y (mgr.25477) 89 : cephadm 1 Reconfiguring daemon osd.0 on smithi007 2023-12-16T07:47:18.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:17 smithi007 bash[65864]: cluster 2023-12-16T07:47:17.127374+0000 mgr.y (mgr.25477) 90 : cluster 0 pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:18.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:17 smithi007 bash[64479]: audit 2023-12-16T07:47:16.889936+0000 mon.a (mon.0) 157 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:18.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:17 smithi007 bash[64479]: audit 2023-12-16T07:47:16.899038+0000 mon.a (mon.0) 158 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:18.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:17 smithi007 bash[64479]: cephadm 2023-12-16T07:47:16.918457+0000 mgr.y (mgr.25477) 88 : cephadm 1 Reconfiguring osd.0 (monmap changed)... 2023-12-16T07:47:18.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:17 smithi007 bash[64479]: audit 2023-12-16T07:47:16.919043+0000 mon.a (mon.0) 159 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-16T07:47:18.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:17 smithi007 bash[64479]: audit 2023-12-16T07:47:16.920660+0000 mon.a (mon.0) 160 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:18.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:17 smithi007 bash[64479]: cephadm 2023-12-16T07:47:16.925682+0000 mgr.y (mgr.25477) 89 : cephadm 1 Reconfiguring daemon osd.0 on smithi007 2023-12-16T07:47:18.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:17 smithi007 bash[64479]: cluster 2023-12-16T07:47:17.127374+0000 mgr.y (mgr.25477) 90 : cluster 0 pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:18.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:47:18 smithi169 bash[49087]: ts=2023-12-16T07:47:18.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:47:20.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:47:20] "GET /metrics HTTP/1.1" 200 38169 "" "Prometheus/2.33.4" 2023-12-16T07:47:20.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[65864]: cluster 2023-12-16T07:47:19.128978+0000 mgr.y (mgr.25477) 91 : cluster 0 pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:20.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[65864]: audit 2023-12-16T07:47:19.405969+0000 mon.a (mon.0) 161 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:20.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[65864]: audit 2023-12-16T07:47:19.415801+0000 mon.a (mon.0) 162 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:20.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[65864]: cephadm 2023-12-16T07:47:19.444881+0000 mgr.y (mgr.25477) 92 : cephadm 1 Reconfiguring osd.1 (monmap changed)... 2023-12-16T07:47:20.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[65864]: audit 2023-12-16T07:47:19.445611+0000 mon.a (mon.0) 163 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-16T07:47:20.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[65864]: audit 2023-12-16T07:47:19.447538+0000 mon.a (mon.0) 164 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:20.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[65864]: cephadm 2023-12-16T07:47:19.453217+0000 mgr.y (mgr.25477) 93 : cephadm 1 Reconfiguring daemon osd.1 on smithi007 2023-12-16T07:47:20.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[64479]: cluster 2023-12-16T07:47:19.128978+0000 mgr.y (mgr.25477) 91 : cluster 0 pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:20.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[64479]: audit 2023-12-16T07:47:19.405969+0000 mon.a (mon.0) 161 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:20.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[64479]: audit 2023-12-16T07:47:19.415801+0000 mon.a (mon.0) 162 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:20.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[64479]: cephadm 2023-12-16T07:47:19.444881+0000 mgr.y (mgr.25477) 92 : cephadm 1 Reconfiguring osd.1 (monmap changed)... 2023-12-16T07:47:20.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[64479]: audit 2023-12-16T07:47:19.445611+0000 mon.a (mon.0) 163 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-16T07:47:20.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[64479]: audit 2023-12-16T07:47:19.447538+0000 mon.a (mon.0) 164 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:20.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:20 smithi007 bash[64479]: cephadm 2023-12-16T07:47:19.453217+0000 mgr.y (mgr.25477) 93 : cephadm 1 Reconfiguring daemon osd.1 on smithi007 2023-12-16T07:47:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:20 smithi169 bash[51638]: cluster 2023-12-16T07:47:19.128978+0000 mgr.y (mgr.25477) 91 : cluster 0 pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:20 smithi169 bash[51638]: audit 2023-12-16T07:47:19.405969+0000 mon.a (mon.0) 161 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:20 smithi169 bash[51638]: audit 2023-12-16T07:47:19.415801+0000 mon.a (mon.0) 162 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:20 smithi169 bash[51638]: cephadm 2023-12-16T07:47:19.444881+0000 mgr.y (mgr.25477) 92 : cephadm 1 Reconfiguring osd.1 (monmap changed)... 2023-12-16T07:47:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:20 smithi169 bash[51638]: audit 2023-12-16T07:47:19.445611+0000 mon.a (mon.0) 163 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-16T07:47:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:20 smithi169 bash[51638]: audit 2023-12-16T07:47:19.447538+0000 mon.a (mon.0) 164 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:20 smithi169 bash[51638]: cephadm 2023-12-16T07:47:19.453217+0000 mgr.y (mgr.25477) 93 : cephadm 1 Reconfiguring daemon osd.1 on smithi007 2023-12-16T07:47:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:22 smithi169 bash[51638]: cluster 2023-12-16T07:47:21.129823+0000 mgr.y (mgr.25477) 94 : cluster 0 pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:22 smithi169 bash[51638]: audit 2023-12-16T07:47:21.871021+0000 mon.a (mon.0) 165 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:22 smithi169 bash[51638]: audit 2023-12-16T07:47:21.880418+0000 mon.a (mon.0) 166 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:22 smithi169 bash[51638]: audit 2023-12-16T07:47:21.923728+0000 mon.a (mon.0) 167 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-16T07:47:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:22 smithi169 bash[51638]: audit 2023-12-16T07:47:21.925514+0000 mon.a (mon.0) 168 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:22 smithi007 bash[64479]: cluster 2023-12-16T07:47:21.129823+0000 mgr.y (mgr.25477) 94 : cluster 0 pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:22 smithi007 bash[64479]: audit 2023-12-16T07:47:21.871021+0000 mon.a (mon.0) 165 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:22 smithi007 bash[64479]: audit 2023-12-16T07:47:21.880418+0000 mon.a (mon.0) 166 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:22.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:22 smithi007 bash[64479]: audit 2023-12-16T07:47:21.923728+0000 mon.a (mon.0) 167 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-16T07:47:22.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:22 smithi007 bash[64479]: audit 2023-12-16T07:47:21.925514+0000 mon.a (mon.0) 168 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:22 smithi007 bash[65864]: cluster 2023-12-16T07:47:21.129823+0000 mgr.y (mgr.25477) 94 : cluster 0 pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:22 smithi007 bash[65864]: audit 2023-12-16T07:47:21.871021+0000 mon.a (mon.0) 165 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:22 smithi007 bash[65864]: audit 2023-12-16T07:47:21.880418+0000 mon.a (mon.0) 166 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:22 smithi007 bash[65864]: audit 2023-12-16T07:47:21.923728+0000 mon.a (mon.0) 167 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-16T07:47:22.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:22 smithi007 bash[65864]: audit 2023-12-16T07:47:21.925514+0000 mon.a (mon.0) 168 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:23.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:23 smithi169 bash[51638]: cephadm 2023-12-16T07:47:21.923038+0000 mgr.y (mgr.25477) 95 : cephadm 1 Reconfiguring osd.2 (monmap changed)... 2023-12-16T07:47:23.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:23 smithi169 bash[51638]: cephadm 2023-12-16T07:47:21.931265+0000 mgr.y (mgr.25477) 96 : cephadm 1 Reconfiguring daemon osd.2 on smithi007 2023-12-16T07:47:23.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:23 smithi169 bash[51638]: audit 2023-12-16T07:47:23.200938+0000 mon.a (mon.0) 169 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:47:23.741 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:23 smithi007 bash[64479]: cephadm 2023-12-16T07:47:21.923038+0000 mgr.y (mgr.25477) 95 : cephadm 1 Reconfiguring osd.2 (monmap changed)... 2023-12-16T07:47:23.742 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:23 smithi007 bash[64479]: cephadm 2023-12-16T07:47:21.931265+0000 mgr.y (mgr.25477) 96 : cephadm 1 Reconfiguring daemon osd.2 on smithi007 2023-12-16T07:47:23.742 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:23 smithi007 bash[64479]: audit 2023-12-16T07:47:23.200938+0000 mon.a (mon.0) 169 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:47:23.742 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:23 smithi007 bash[65864]: cephadm 2023-12-16T07:47:21.923038+0000 mgr.y (mgr.25477) 95 : cephadm 1 Reconfiguring osd.2 (monmap changed)... 2023-12-16T07:47:23.742 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:23 smithi007 bash[65864]: cephadm 2023-12-16T07:47:21.931265+0000 mgr.y (mgr.25477) 96 : cephadm 1 Reconfiguring daemon osd.2 on smithi007 2023-12-16T07:47:23.742 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:23 smithi007 bash[65864]: audit 2023-12-16T07:47:23.200938+0000 mon.a (mon.0) 169 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:47:24.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:24 smithi169 bash[51638]: cluster 2023-12-16T07:47:23.130572+0000 mgr.y (mgr.25477) 97 : cluster 0 pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:24.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:24 smithi007 bash[65864]: cluster 2023-12-16T07:47:23.130572+0000 mgr.y (mgr.25477) 97 : cluster 0 pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:24.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:24 smithi007 bash[64479]: cluster 2023-12-16T07:47:23.130572+0000 mgr.y (mgr.25477) 97 : cluster 0 pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:25.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:25 smithi007 bash[64479]: audit 2023-12-16T07:47:24.167298+0000 mgr.y (mgr.25477) 98 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:25.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:25 smithi007 bash[64479]: audit 2023-12-16T07:47:24.432639+0000 mon.a (mon.0) 170 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:25.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:25 smithi007 bash[64479]: audit 2023-12-16T07:47:24.440836+0000 mon.a (mon.0) 171 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:25.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:25 smithi007 bash[64479]: cephadm 2023-12-16T07:47:24.449577+0000 mgr.y (mgr.25477) 99 : cephadm 1 Reconfiguring osd.3 (monmap changed)... 2023-12-16T07:47:25.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:25 smithi007 bash[64479]: audit 2023-12-16T07:47:24.450141+0000 mon.a (mon.0) 172 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-16T07:47:25.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:25 smithi007 bash[64479]: audit 2023-12-16T07:47:24.451409+0000 mon.a (mon.0) 173 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:25.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:25 smithi007 bash[64479]: cephadm 2023-12-16T07:47:24.454900+0000 mgr.y (mgr.25477) 100 : cephadm 1 Reconfiguring daemon osd.3 on smithi007 2023-12-16T07:47:25.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:25 smithi007 bash[65864]: audit 2023-12-16T07:47:24.167298+0000 mgr.y (mgr.25477) 98 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:25.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:25 smithi007 bash[65864]: audit 2023-12-16T07:47:24.432639+0000 mon.a (mon.0) 170 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:25.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:25 smithi007 bash[65864]: audit 2023-12-16T07:47:24.440836+0000 mon.a (mon.0) 171 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:25.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:25 smithi007 bash[65864]: cephadm 2023-12-16T07:47:24.449577+0000 mgr.y (mgr.25477) 99 : cephadm 1 Reconfiguring osd.3 (monmap changed)... 2023-12-16T07:47:25.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:25 smithi007 bash[65864]: audit 2023-12-16T07:47:24.450141+0000 mon.a (mon.0) 172 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-16T07:47:25.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:25 smithi007 bash[65864]: audit 2023-12-16T07:47:24.451409+0000 mon.a (mon.0) 173 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:25.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:25 smithi007 bash[65864]: cephadm 2023-12-16T07:47:24.454900+0000 mgr.y (mgr.25477) 100 : cephadm 1 Reconfiguring daemon osd.3 on smithi007 2023-12-16T07:47:25.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:25 smithi169 bash[51638]: audit 2023-12-16T07:47:24.167298+0000 mgr.y (mgr.25477) 98 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:25.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:25 smithi169 bash[51638]: audit 2023-12-16T07:47:24.432639+0000 mon.a (mon.0) 170 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:25.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:25 smithi169 bash[51638]: audit 2023-12-16T07:47:24.440836+0000 mon.a (mon.0) 171 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:25.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:25 smithi169 bash[51638]: cephadm 2023-12-16T07:47:24.449577+0000 mgr.y (mgr.25477) 99 : cephadm 1 Reconfiguring osd.3 (monmap changed)... 2023-12-16T07:47:25.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:25 smithi169 bash[51638]: audit 2023-12-16T07:47:24.450141+0000 mon.a (mon.0) 172 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-16T07:47:25.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:25 smithi169 bash[51638]: audit 2023-12-16T07:47:24.451409+0000 mon.a (mon.0) 173 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:25.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:25 smithi169 bash[51638]: cephadm 2023-12-16T07:47:24.454900+0000 mgr.y (mgr.25477) 100 : cephadm 1 Reconfiguring daemon osd.3 on smithi007 2023-12-16T07:47:26.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:26 smithi007 bash[64479]: cluster 2023-12-16T07:47:25.132225+0000 mgr.y (mgr.25477) 101 : cluster 0 pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:26.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:26 smithi007 bash[65864]: cluster 2023-12-16T07:47:25.132225+0000 mgr.y (mgr.25477) 101 : cluster 0 pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:26.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:26 smithi169 bash[51638]: cluster 2023-12-16T07:47:25.132225+0000 mgr.y (mgr.25477) 101 : cluster 0 pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:28.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:28 smithi007 bash[64479]: audit 2023-12-16T07:47:27.013914+0000 mon.a (mon.0) 174 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:28.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:28 smithi007 bash[64479]: audit 2023-12-16T07:47:27.024220+0000 mon.a (mon.0) 175 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:28.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:28 smithi007 bash[64479]: cephadm 2023-12-16T07:47:27.076130+0000 mgr.y (mgr.25477) 102 : cephadm 1 Reconfiguring rgw.foo.smithi007.mglnhl (monmap changed)... 2023-12-16T07:47:28.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:28 smithi007 bash[64479]: audit 2023-12-16T07:47:27.076938+0000 mon.a (mon.0) 176 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.mglnhl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:47:28.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:28 smithi007 bash[64479]: audit 2023-12-16T07:47:27.081050+0000 mon.a (mon.0) 177 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:28.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:28 smithi007 bash[64479]: cephadm 2023-12-16T07:47:27.083316+0000 mgr.y (mgr.25477) 103 : cephadm 1 Reconfiguring daemon rgw.foo.smithi007.mglnhl on smithi007 2023-12-16T07:47:28.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:28 smithi007 bash[64479]: cluster 2023-12-16T07:47:27.132971+0000 mgr.y (mgr.25477) 104 : cluster 0 pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:28.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:28 smithi007 bash[65864]: audit 2023-12-16T07:47:27.013914+0000 mon.a (mon.0) 174 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:28.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:28 smithi007 bash[65864]: audit 2023-12-16T07:47:27.024220+0000 mon.a (mon.0) 175 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:28.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:28 smithi007 bash[65864]: cephadm 2023-12-16T07:47:27.076130+0000 mgr.y (mgr.25477) 102 : cephadm 1 Reconfiguring rgw.foo.smithi007.mglnhl (monmap changed)... 2023-12-16T07:47:28.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:28 smithi007 bash[65864]: audit 2023-12-16T07:47:27.076938+0000 mon.a (mon.0) 176 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.mglnhl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:47:28.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:28 smithi007 bash[65864]: audit 2023-12-16T07:47:27.081050+0000 mon.a (mon.0) 177 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:28.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:28 smithi007 bash[65864]: cephadm 2023-12-16T07:47:27.083316+0000 mgr.y (mgr.25477) 103 : cephadm 1 Reconfiguring daemon rgw.foo.smithi007.mglnhl on smithi007 2023-12-16T07:47:28.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:28 smithi007 bash[65864]: cluster 2023-12-16T07:47:27.132971+0000 mgr.y (mgr.25477) 104 : cluster 0 pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:28.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:28 smithi169 bash[51638]: audit 2023-12-16T07:47:27.013914+0000 mon.a (mon.0) 174 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:28.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:28 smithi169 bash[51638]: audit 2023-12-16T07:47:27.024220+0000 mon.a (mon.0) 175 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:28.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:28 smithi169 bash[51638]: cephadm 2023-12-16T07:47:27.076130+0000 mgr.y (mgr.25477) 102 : cephadm 1 Reconfiguring rgw.foo.smithi007.mglnhl (monmap changed)... 2023-12-16T07:47:28.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:28 smithi169 bash[51638]: audit 2023-12-16T07:47:27.076938+0000 mon.a (mon.0) 176 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.mglnhl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:47:28.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:28 smithi169 bash[51638]: audit 2023-12-16T07:47:27.081050+0000 mon.a (mon.0) 177 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:28.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:28 smithi169 bash[51638]: cephadm 2023-12-16T07:47:27.083316+0000 mgr.y (mgr.25477) 103 : cephadm 1 Reconfiguring daemon rgw.foo.smithi007.mglnhl on smithi007 2023-12-16T07:47:28.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:28 smithi169 bash[51638]: cluster 2023-12-16T07:47:27.132971+0000 mgr.y (mgr.25477) 104 : cluster 0 pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:28.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:47:28 smithi169 bash[49087]: ts=2023-12-16T07:47:28.662Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:47:30.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:47:30] "GET /metrics HTTP/1.1" 200 38173 "" "Prometheus/2.33.4" 2023-12-16T07:47:30.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[65864]: cluster 2023-12-16T07:47:29.134657+0000 mgr.y (mgr.25477) 105 : cluster 0 pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[65864]: audit 2023-12-16T07:47:29.481146+0000 mon.a (mon.0) 178 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[65864]: audit 2023-12-16T07:47:29.490042+0000 mon.a (mon.0) 179 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[65864]: cephadm 2023-12-16T07:47:29.502988+0000 mgr.y (mgr.25477) 106 : cephadm 1 Reconfiguring rgw.smpl.smithi007.tizuzt (monmap changed)... 2023-12-16T07:47:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[65864]: audit 2023-12-16T07:47:29.503635+0000 mon.a (mon.0) 180 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.tizuzt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:47:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[65864]: audit 2023-12-16T07:47:29.506546+0000 mon.a (mon.0) 181 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[65864]: cephadm 2023-12-16T07:47:29.508174+0000 mgr.y (mgr.25477) 107 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi007.tizuzt on smithi007 2023-12-16T07:47:30.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[64479]: cluster 2023-12-16T07:47:29.134657+0000 mgr.y (mgr.25477) 105 : cluster 0 pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:30.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[64479]: audit 2023-12-16T07:47:29.481146+0000 mon.a (mon.0) 178 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:30.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[64479]: audit 2023-12-16T07:47:29.490042+0000 mon.a (mon.0) 179 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:30.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[64479]: cephadm 2023-12-16T07:47:29.502988+0000 mgr.y (mgr.25477) 106 : cephadm 1 Reconfiguring rgw.smpl.smithi007.tizuzt (monmap changed)... 2023-12-16T07:47:30.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[64479]: audit 2023-12-16T07:47:29.503635+0000 mon.a (mon.0) 180 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.tizuzt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:47:30.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[64479]: audit 2023-12-16T07:47:29.506546+0000 mon.a (mon.0) 181 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:30.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:30 smithi007 bash[64479]: cephadm 2023-12-16T07:47:29.508174+0000 mgr.y (mgr.25477) 107 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi007.tizuzt on smithi007 2023-12-16T07:47:30.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:30 smithi169 bash[51638]: cluster 2023-12-16T07:47:29.134657+0000 mgr.y (mgr.25477) 105 : cluster 0 pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:30.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:30 smithi169 bash[51638]: audit 2023-12-16T07:47:29.481146+0000 mon.a (mon.0) 178 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:30.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:30 smithi169 bash[51638]: audit 2023-12-16T07:47:29.490042+0000 mon.a (mon.0) 179 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:30.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:30 smithi169 bash[51638]: cephadm 2023-12-16T07:47:29.502988+0000 mgr.y (mgr.25477) 106 : cephadm 1 Reconfiguring rgw.smpl.smithi007.tizuzt (monmap changed)... 2023-12-16T07:47:30.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:30 smithi169 bash[51638]: audit 2023-12-16T07:47:29.503635+0000 mon.a (mon.0) 180 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.tizuzt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:47:30.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:30 smithi169 bash[51638]: audit 2023-12-16T07:47:29.506546+0000 mon.a (mon.0) 181 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:30.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:30 smithi169 bash[51638]: cephadm 2023-12-16T07:47:29.508174+0000 mgr.y (mgr.25477) 107 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi007.tizuzt on smithi007 2023-12-16T07:47:32.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:32 smithi169 bash[51638]: cluster 2023-12-16T07:47:31.135373+0000 mgr.y (mgr.25477) 108 : cluster 0 pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:32.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:32 smithi169 bash[51638]: audit 2023-12-16T07:47:31.869899+0000 mon.a (mon.0) 182 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:32.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:32 smithi169 bash[51638]: audit 2023-12-16T07:47:31.877569+0000 mon.a (mon.0) 183 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:32.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:32 smithi169 bash[51638]: audit 2023-12-16T07:47:31.929046+0000 mon.a (mon.0) 184 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:47:32.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:32 smithi169 bash[51638]: audit 2023-12-16T07:47:31.930780+0000 mon.a (mon.0) 185 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:47:32.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:32 smithi169 bash[51638]: audit 2023-12-16T07:47:31.932450+0000 mon.a (mon.0) 186 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:32.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:32 smithi007 bash[64479]: cluster 2023-12-16T07:47:31.135373+0000 mgr.y (mgr.25477) 108 : cluster 0 pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:32.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:32 smithi007 bash[64479]: audit 2023-12-16T07:47:31.869899+0000 mon.a (mon.0) 182 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:32.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:32 smithi007 bash[64479]: audit 2023-12-16T07:47:31.877569+0000 mon.a (mon.0) 183 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:32.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:32 smithi007 bash[64479]: audit 2023-12-16T07:47:31.929046+0000 mon.a (mon.0) 184 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:47:32.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:32 smithi007 bash[64479]: audit 2023-12-16T07:47:31.930780+0000 mon.a (mon.0) 185 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:47:32.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:32 smithi007 bash[64479]: audit 2023-12-16T07:47:31.932450+0000 mon.a (mon.0) 186 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:32.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:32 smithi007 bash[65864]: cluster 2023-12-16T07:47:31.135373+0000 mgr.y (mgr.25477) 108 : cluster 0 pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:32.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:32 smithi007 bash[65864]: audit 2023-12-16T07:47:31.869899+0000 mon.a (mon.0) 182 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:32.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:32 smithi007 bash[65864]: audit 2023-12-16T07:47:31.877569+0000 mon.a (mon.0) 183 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:32.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:32 smithi007 bash[65864]: audit 2023-12-16T07:47:31.929046+0000 mon.a (mon.0) 184 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-12-16T07:47:32.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:32 smithi007 bash[65864]: audit 2023-12-16T07:47:31.930780+0000 mon.a (mon.0) 185 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-12-16T07:47:32.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:32 smithi007 bash[65864]: audit 2023-12-16T07:47:31.932450+0000 mon.a (mon.0) 186 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:33.660 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:33 smithi169 bash[51638]: cephadm 2023-12-16T07:47:31.928351+0000 mgr.y (mgr.25477) 109 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2023-12-16T07:47:33.660 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:33 smithi169 bash[51638]: cephadm 2023-12-16T07:47:31.934537+0000 mgr.y (mgr.25477) 110 : cephadm 1 Reconfiguring daemon mon.b on smithi169 2023-12-16T07:47:33.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:33 smithi007 bash[65864]: cephadm 2023-12-16T07:47:31.928351+0000 mgr.y (mgr.25477) 109 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2023-12-16T07:47:33.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:33 smithi007 bash[65864]: cephadm 2023-12-16T07:47:31.934537+0000 mgr.y (mgr.25477) 110 : cephadm 1 Reconfiguring daemon mon.b on smithi169 2023-12-16T07:47:33.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:33 smithi007 bash[64479]: cephadm 2023-12-16T07:47:31.928351+0000 mgr.y (mgr.25477) 109 : cephadm 1 Reconfiguring mon.b (monmap changed)... 2023-12-16T07:47:33.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:33 smithi007 bash[64479]: cephadm 2023-12-16T07:47:31.934537+0000 mgr.y (mgr.25477) 110 : cephadm 1 Reconfiguring daemon mon.b on smithi169 2023-12-16T07:47:34.503 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:34 smithi007 bash[64479]: cluster 2023-12-16T07:47:33.136154+0000 mgr.y (mgr.25477) 111 : cluster 0 pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:34.503 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:34 smithi007 bash[64479]: audit 2023-12-16T07:47:34.237627+0000 mon.a (mon.0) 187 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:34.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:34 smithi007 bash[64479]: audit 2023-12-16T07:47:34.249992+0000 mon.a (mon.0) 188 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:34.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:34 smithi007 bash[64479]: audit 2023-12-16T07:47:34.256009+0000 mon.a (mon.0) 189 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:47:34.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:34 smithi007 bash[64479]: audit 2023-12-16T07:47:34.258050+0000 mon.a (mon.0) 190 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:47:34.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:34 smithi007 bash[64479]: audit 2023-12-16T07:47:34.259674+0000 mon.a (mon.0) 191 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:34.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:34 smithi007 bash[65864]: cluster 2023-12-16T07:47:33.136154+0000 mgr.y (mgr.25477) 111 : cluster 0 pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:34.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:34 smithi007 bash[65864]: audit 2023-12-16T07:47:34.237627+0000 mon.a (mon.0) 187 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:34.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:34 smithi007 bash[65864]: audit 2023-12-16T07:47:34.249992+0000 mon.a (mon.0) 188 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:34.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:34 smithi007 bash[65864]: audit 2023-12-16T07:47:34.256009+0000 mon.a (mon.0) 189 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:47:34.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:34 smithi007 bash[65864]: audit 2023-12-16T07:47:34.258050+0000 mon.a (mon.0) 190 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:47:34.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:34 smithi007 bash[65864]: audit 2023-12-16T07:47:34.259674+0000 mon.a (mon.0) 191 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:34.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:34 smithi169 bash[51638]: cluster 2023-12-16T07:47:33.136154+0000 mgr.y (mgr.25477) 111 : cluster 0 pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:34.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:34 smithi169 bash[51638]: audit 2023-12-16T07:47:34.237627+0000 mon.a (mon.0) 187 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:34.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:34 smithi169 bash[51638]: audit 2023-12-16T07:47:34.249992+0000 mon.a (mon.0) 188 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:34.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:34 smithi169 bash[51638]: audit 2023-12-16T07:47:34.256009+0000 mon.a (mon.0) 189 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-12-16T07:47:34.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:34 smithi169 bash[51638]: audit 2023-12-16T07:47:34.258050+0000 mon.a (mon.0) 190 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-12-16T07:47:34.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:34 smithi169 bash[51638]: audit 2023-12-16T07:47:34.259674+0000 mon.a (mon.0) 191 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:35.662 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:47:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:35 smithi169 bash[51638]: audit 2023-12-16T07:47:34.175236+0000 mgr.y (mgr.25477) 112 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:35 smithi169 bash[51638]: cephadm 2023-12-16T07:47:34.255323+0000 mgr.y (mgr.25477) 113 : cephadm 1 Reconfiguring mgr.x (monmap changed)... 2023-12-16T07:47:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:35 smithi169 bash[51638]: cephadm 2023-12-16T07:47:34.261577+0000 mgr.y (mgr.25477) 114 : cephadm 1 Reconfiguring daemon mgr.x on smithi169 2023-12-16T07:47:35.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:35 smithi007 bash[64479]: audit 2023-12-16T07:47:34.175236+0000 mgr.y (mgr.25477) 112 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:35.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:35 smithi007 bash[64479]: cephadm 2023-12-16T07:47:34.255323+0000 mgr.y (mgr.25477) 113 : cephadm 1 Reconfiguring mgr.x (monmap changed)... 2023-12-16T07:47:35.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:35 smithi007 bash[64479]: cephadm 2023-12-16T07:47:34.261577+0000 mgr.y (mgr.25477) 114 : cephadm 1 Reconfiguring daemon mgr.x on smithi169 2023-12-16T07:47:35.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:35 smithi007 bash[65864]: audit 2023-12-16T07:47:34.175236+0000 mgr.y (mgr.25477) 112 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:35.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:35 smithi007 bash[65864]: cephadm 2023-12-16T07:47:34.255323+0000 mgr.y (mgr.25477) 113 : cephadm 1 Reconfiguring mgr.x (monmap changed)... 2023-12-16T07:47:35.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:35 smithi007 bash[65864]: cephadm 2023-12-16T07:47:34.261577+0000 mgr.y (mgr.25477) 114 : cephadm 1 Reconfiguring daemon mgr.x on smithi169 2023-12-16T07:47:36.441 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:47:36.442 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (4m) 69s ago 11m 16.4M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:47:36.442 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (3m) 38s ago 10m 43.6M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:47:36.442 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (2m) 69s ago 8m 45.7M - 3.5 e1d6a67b021e 8d8009d0c83f 2023-12-16T07:47:36.442 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283,8765 running (2m) 38s ago 17m 433M - 19.0.0-191-g6d933094 a29c258b5d6f 5295ba35c2cd 2023-12-16T07:47:36.442 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (3m) 69s ago 19m 478M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:47:36.442 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (113s) 69s ago 19m 39.0M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 5a266b45f3ce 2023-12-16T07:47:36.442 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (51s) 38s ago 17m 23.6M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1fc9bf864dda 2023-12-16T07:47:36.442 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (77s) 69s ago 17m 22.5M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1dc91c31fc27 2023-12-16T07:47:36.442 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (4m) 69s ago 11m 11.0M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:47:36.442 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (3m) 38s ago 11m 11.3M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:47:36.442 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (16m) 69s ago 16m 58.3M 2782M 17.2.0 e1d6a67b021e dbc4d9538238 2023-12-16T07:47:36.442 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (16m) 69s ago 16m 59.2M 2782M 17.2.0 e1d6a67b021e 14637e0f389c 2023-12-16T07:47:36.443 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (15m) 69s ago 15m 57.9M 2782M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:47:36.443 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (14m) 69s ago 14m 57.1M 2782M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:47:36.443 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (14m) 38s ago 14m 58.5M 3037M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:47:36.443 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (13m) 38s ago 13m 55.3M 3037M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:47:36.443 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (12m) 38s ago 13m 55.8M 3037M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:47:36.443 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (12m) 38s ago 12m 56.9M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:47:36.443 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (2m) 38s ago 11m 59.0M - 2.33.4 514e6a882f6e 2652ce1b8540 2023-12-16T07:47:36.443 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (9m) 69s ago 9m 87.0M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:47:36.443 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (9m) 38s ago 9m 87.4M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:47:36.443 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (9m) 69s ago 9m 86.8M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:47:36.443 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (9m) 38s ago 9m 87.0M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:47:36.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:36 smithi007 bash[64479]: cluster 2023-12-16T07:47:35.137840+0000 mgr.y (mgr.25477) 115 : cluster 0 pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:36.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:36 smithi007 bash[64479]: audit 2023-12-16T07:47:36.294634+0000 mon.a (mon.0) 192 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:36.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:36 smithi007 bash[64479]: audit 2023-12-16T07:47:36.304148+0000 mon.a (mon.0) 193 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:36.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:36 smithi007 bash[64479]: audit 2023-12-16T07:47:36.330799+0000 mon.a (mon.0) 194 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-16T07:47:36.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:36 smithi007 bash[64479]: audit 2023-12-16T07:47:36.332177+0000 mon.a (mon.0) 195 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:36.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:36 smithi007 bash[65864]: cluster 2023-12-16T07:47:35.137840+0000 mgr.y (mgr.25477) 115 : cluster 0 pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:36.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:36 smithi007 bash[65864]: audit 2023-12-16T07:47:36.294634+0000 mon.a (mon.0) 192 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:36.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:36 smithi007 bash[65864]: audit 2023-12-16T07:47:36.304148+0000 mon.a (mon.0) 193 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:36.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:36 smithi007 bash[65864]: audit 2023-12-16T07:47:36.330799+0000 mon.a (mon.0) 194 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-16T07:47:36.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:36 smithi007 bash[65864]: audit 2023-12-16T07:47:36.332177+0000 mon.a (mon.0) 195 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:36.908 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:47:36.909 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:47:36.909 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 3 2023-12-16T07:47:36.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:47:36.909 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:47:36.909 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:47:36.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:47:36.909 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:47:36.909 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2023-12-16T07:47:36.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:47:36.909 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:47:36.909 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:47:36.909 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:47:36.909 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:47:36.910 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2023-12-16T07:47:36.910 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 5 2023-12-16T07:47:36.910 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:47:36.910 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:47:36.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:36 smithi169 bash[51638]: cluster 2023-12-16T07:47:35.137840+0000 mgr.y (mgr.25477) 115 : cluster 0 pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:36.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:36 smithi169 bash[51638]: audit 2023-12-16T07:47:36.294634+0000 mon.a (mon.0) 192 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:36.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:36 smithi169 bash[51638]: audit 2023-12-16T07:47:36.304148+0000 mon.a (mon.0) 193 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:36.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:36 smithi169 bash[51638]: audit 2023-12-16T07:47:36.330799+0000 mon.a (mon.0) 194 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-16T07:47:36.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:36 smithi169 bash[51638]: audit 2023-12-16T07:47:36.332177+0000 mon.a (mon.0) 195 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:37.300 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:47:37.300 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:47:37.300 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:47:37.300 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:47:37.300 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2023-12-16T07:47:37.300 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr", 2023-12-16T07:47:37.301 INFO:teuthology.orchestra.run.smithi007.stdout: "mon" 2023-12-16T07:47:37.301 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:47:37.301 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "5/23 daemons upgraded", 2023-12-16T07:47:37.301 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading mon daemons", 2023-12-16T07:47:37.301 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:47:37.301 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:47:37.754 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:47:37.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:37 smithi007 bash[64479]: audit 2023-12-16T07:47:35.643071+0000 mgr.y (mgr.25477) 116 : audit 0 from='client.44173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:37.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:37 smithi007 bash[64479]: audit 2023-12-16T07:47:36.037624+0000 mgr.y (mgr.25477) 117 : audit 0 from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:37.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:37 smithi007 bash[64479]: cephadm 2023-12-16T07:47:36.330200+0000 mgr.y (mgr.25477) 118 : cephadm 1 Reconfiguring osd.4 (monmap changed)... 2023-12-16T07:47:37.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:37 smithi007 bash[64479]: cephadm 2023-12-16T07:47:36.336402+0000 mgr.y (mgr.25477) 119 : cephadm 1 Reconfiguring daemon osd.4 on smithi169 2023-12-16T07:47:37.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:37 smithi007 bash[64479]: audit 2023-12-16T07:47:36.422166+0000 mgr.y (mgr.25477) 120 : audit 0 from='client.44185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:37.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:37 smithi007 bash[64479]: audit 2023-12-16T07:47:36.901349+0000 mon.c (mon.1) 6 : audit 0 from='client.? 172.21.15.7:0/3867193253' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:37.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:37 smithi007 bash[65864]: audit 2023-12-16T07:47:35.643071+0000 mgr.y (mgr.25477) 116 : audit 0 from='client.44173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:37.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:37 smithi007 bash[65864]: audit 2023-12-16T07:47:36.037624+0000 mgr.y (mgr.25477) 117 : audit 0 from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:37.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:37 smithi007 bash[65864]: cephadm 2023-12-16T07:47:36.330200+0000 mgr.y (mgr.25477) 118 : cephadm 1 Reconfiguring osd.4 (monmap changed)... 2023-12-16T07:47:37.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:37 smithi007 bash[65864]: cephadm 2023-12-16T07:47:36.336402+0000 mgr.y (mgr.25477) 119 : cephadm 1 Reconfiguring daemon osd.4 on smithi169 2023-12-16T07:47:37.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:37 smithi007 bash[65864]: audit 2023-12-16T07:47:36.422166+0000 mgr.y (mgr.25477) 120 : audit 0 from='client.44185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:37.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:37 smithi007 bash[65864]: audit 2023-12-16T07:47:36.901349+0000 mon.c (mon.1) 6 : audit 0 from='client.? 172.21.15.7:0/3867193253' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:37.877 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:37 smithi169 bash[51638]: audit 2023-12-16T07:47:35.643071+0000 mgr.y (mgr.25477) 116 : audit 0 from='client.44173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:37.877 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:37 smithi169 bash[51638]: audit 2023-12-16T07:47:36.037624+0000 mgr.y (mgr.25477) 117 : audit 0 from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:37.877 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:37 smithi169 bash[51638]: cephadm 2023-12-16T07:47:36.330200+0000 mgr.y (mgr.25477) 118 : cephadm 1 Reconfiguring osd.4 (monmap changed)... 2023-12-16T07:47:37.878 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:37 smithi169 bash[51638]: cephadm 2023-12-16T07:47:36.336402+0000 mgr.y (mgr.25477) 119 : cephadm 1 Reconfiguring daemon osd.4 on smithi169 2023-12-16T07:47:37.878 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:37 smithi169 bash[51638]: audit 2023-12-16T07:47:36.422166+0000 mgr.y (mgr.25477) 120 : audit 0 from='client.44185 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:37.878 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:37 smithi169 bash[51638]: audit 2023-12-16T07:47:36.901349+0000 mon.c (mon.1) 6 : audit 0 from='client.? 172.21.15.7:0/3867193253' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:38 smithi007 bash[65864]: cluster 2023-12-16T07:47:37.138677+0000 mgr.y (mgr.25477) 121 : cluster 0 pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:38 smithi007 bash[65864]: audit 2023-12-16T07:47:37.296571+0000 mgr.y (mgr.25477) 122 : audit 0 from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:38.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:38 smithi007 bash[65864]: audit 2023-12-16T07:47:37.752360+0000 mon.a (mon.0) 196 : audit 0 from='client.? 172.21.15.7:0/3192433271' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:47:38.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:38 smithi007 bash[65864]: audit 2023-12-16T07:47:38.201489+0000 mon.a (mon.0) 197 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:47:38.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:38 smithi007 bash[64479]: cluster 2023-12-16T07:47:37.138677+0000 mgr.y (mgr.25477) 121 : cluster 0 pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:38.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:38 smithi007 bash[64479]: audit 2023-12-16T07:47:37.296571+0000 mgr.y (mgr.25477) 122 : audit 0 from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:38.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:38 smithi007 bash[64479]: audit 2023-12-16T07:47:37.752360+0000 mon.a (mon.0) 196 : audit 0 from='client.? 172.21.15.7:0/3192433271' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:47:38.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:38 smithi007 bash[64479]: audit 2023-12-16T07:47:38.201489+0000 mon.a (mon.0) 197 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:47:38.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:47:38 smithi169 bash[49087]: ts=2023-12-16T07:47:38.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:47:38.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:38 smithi169 bash[51638]: cluster 2023-12-16T07:47:37.138677+0000 mgr.y (mgr.25477) 121 : cluster 0 pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:38.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:38 smithi169 bash[51638]: audit 2023-12-16T07:47:37.296571+0000 mgr.y (mgr.25477) 122 : audit 0 from='client.34263 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:47:38.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:38 smithi169 bash[51638]: audit 2023-12-16T07:47:37.752360+0000 mon.a (mon.0) 196 : audit 0 from='client.? 172.21.15.7:0/3192433271' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:47:38.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:38 smithi169 bash[51638]: audit 2023-12-16T07:47:38.201489+0000 mon.a (mon.0) 197 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:47:39.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:39 smithi007 bash[65864]: audit 2023-12-16T07:47:38.537517+0000 mon.a (mon.0) 198 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:39.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:39 smithi007 bash[65864]: audit 2023-12-16T07:47:38.546753+0000 mon.a (mon.0) 199 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:39.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:39 smithi007 bash[65864]: cephadm 2023-12-16T07:47:38.556763+0000 mgr.y (mgr.25477) 123 : cephadm 1 Reconfiguring osd.5 (monmap changed)... 2023-12-16T07:47:39.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:39 smithi007 bash[65864]: audit 2023-12-16T07:47:38.557496+0000 mon.a (mon.0) 200 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-16T07:47:39.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:39 smithi007 bash[65864]: audit 2023-12-16T07:47:38.558893+0000 mon.a (mon.0) 201 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:39.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:39 smithi007 bash[65864]: cephadm 2023-12-16T07:47:38.563509+0000 mgr.y (mgr.25477) 124 : cephadm 1 Reconfiguring daemon osd.5 on smithi169 2023-12-16T07:47:39.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:39 smithi007 bash[65864]: cluster 2023-12-16T07:47:39.140424+0000 mgr.y (mgr.25477) 125 : cluster 0 pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:39.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:39 smithi007 bash[64479]: audit 2023-12-16T07:47:38.537517+0000 mon.a (mon.0) 198 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:39.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:39 smithi007 bash[64479]: audit 2023-12-16T07:47:38.546753+0000 mon.a (mon.0) 199 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:39.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:39 smithi007 bash[64479]: cephadm 2023-12-16T07:47:38.556763+0000 mgr.y (mgr.25477) 123 : cephadm 1 Reconfiguring osd.5 (monmap changed)... 2023-12-16T07:47:39.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:39 smithi007 bash[64479]: audit 2023-12-16T07:47:38.557496+0000 mon.a (mon.0) 200 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-16T07:47:39.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:39 smithi007 bash[64479]: audit 2023-12-16T07:47:38.558893+0000 mon.a (mon.0) 201 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:39.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:39 smithi007 bash[64479]: cephadm 2023-12-16T07:47:38.563509+0000 mgr.y (mgr.25477) 124 : cephadm 1 Reconfiguring daemon osd.5 on smithi169 2023-12-16T07:47:39.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:39 smithi007 bash[64479]: cluster 2023-12-16T07:47:39.140424+0000 mgr.y (mgr.25477) 125 : cluster 0 pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:39.823 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:39 smithi169 bash[51638]: audit 2023-12-16T07:47:38.537517+0000 mon.a (mon.0) 198 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:39.823 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:39 smithi169 bash[51638]: audit 2023-12-16T07:47:38.546753+0000 mon.a (mon.0) 199 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:39.823 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:39 smithi169 bash[51638]: cephadm 2023-12-16T07:47:38.556763+0000 mgr.y (mgr.25477) 123 : cephadm 1 Reconfiguring osd.5 (monmap changed)... 2023-12-16T07:47:39.824 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:39 smithi169 bash[51638]: audit 2023-12-16T07:47:38.557496+0000 mon.a (mon.0) 200 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-16T07:47:39.824 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:39 smithi169 bash[51638]: audit 2023-12-16T07:47:38.558893+0000 mon.a (mon.0) 201 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:39.824 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:39 smithi169 bash[51638]: cephadm 2023-12-16T07:47:38.563509+0000 mgr.y (mgr.25477) 124 : cephadm 1 Reconfiguring daemon osd.5 on smithi169 2023-12-16T07:47:39.824 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:39 smithi169 bash[51638]: cluster 2023-12-16T07:47:39.140424+0000 mgr.y (mgr.25477) 125 : cluster 0 pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:40.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:47:40 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:47:40] "GET /metrics HTTP/1.1" 200 38169 "" "Prometheus/2.33.4" 2023-12-16T07:47:41.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:41 smithi169 bash[51638]: audit 2023-12-16T07:47:40.691706+0000 mon.a (mon.0) 202 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:41.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:41 smithi169 bash[51638]: audit 2023-12-16T07:47:40.702756+0000 mon.a (mon.0) 203 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:41.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:41 smithi169 bash[51638]: cephadm 2023-12-16T07:47:40.732539+0000 mgr.y (mgr.25477) 126 : cephadm 1 Reconfiguring osd.6 (monmap changed)... 2023-12-16T07:47:41.955 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:41 smithi169 bash[51638]: audit 2023-12-16T07:47:40.733242+0000 mon.a (mon.0) 204 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-12-16T07:47:41.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:41 smithi169 bash[51638]: audit 2023-12-16T07:47:40.735143+0000 mon.a (mon.0) 205 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:41.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:41 smithi169 bash[51638]: cephadm 2023-12-16T07:47:40.740681+0000 mgr.y (mgr.25477) 127 : cephadm 1 Reconfiguring daemon osd.6 on smithi169 2023-12-16T07:47:41.956 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:41 smithi169 bash[51638]: cluster 2023-12-16T07:47:41.141238+0000 mgr.y (mgr.25477) 128 : cluster 0 pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:42.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:41 smithi007 bash[65864]: audit 2023-12-16T07:47:40.691706+0000 mon.a (mon.0) 202 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:42.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:41 smithi007 bash[65864]: audit 2023-12-16T07:47:40.702756+0000 mon.a (mon.0) 203 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:42.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:41 smithi007 bash[65864]: cephadm 2023-12-16T07:47:40.732539+0000 mgr.y (mgr.25477) 126 : cephadm 1 Reconfiguring osd.6 (monmap changed)... 2023-12-16T07:47:42.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:41 smithi007 bash[65864]: audit 2023-12-16T07:47:40.733242+0000 mon.a (mon.0) 204 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-12-16T07:47:42.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:41 smithi007 bash[65864]: audit 2023-12-16T07:47:40.735143+0000 mon.a (mon.0) 205 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:42.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:41 smithi007 bash[65864]: cephadm 2023-12-16T07:47:40.740681+0000 mgr.y (mgr.25477) 127 : cephadm 1 Reconfiguring daemon osd.6 on smithi169 2023-12-16T07:47:42.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:41 smithi007 bash[65864]: cluster 2023-12-16T07:47:41.141238+0000 mgr.y (mgr.25477) 128 : cluster 0 pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:42.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:41 smithi007 bash[64479]: audit 2023-12-16T07:47:40.691706+0000 mon.a (mon.0) 202 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:42.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:41 smithi007 bash[64479]: audit 2023-12-16T07:47:40.702756+0000 mon.a (mon.0) 203 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:42.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:41 smithi007 bash[64479]: cephadm 2023-12-16T07:47:40.732539+0000 mgr.y (mgr.25477) 126 : cephadm 1 Reconfiguring osd.6 (monmap changed)... 2023-12-16T07:47:42.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:41 smithi007 bash[64479]: audit 2023-12-16T07:47:40.733242+0000 mon.a (mon.0) 204 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-12-16T07:47:42.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:41 smithi007 bash[64479]: audit 2023-12-16T07:47:40.735143+0000 mon.a (mon.0) 205 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:42.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:41 smithi007 bash[64479]: cephadm 2023-12-16T07:47:40.740681+0000 mgr.y (mgr.25477) 127 : cephadm 1 Reconfiguring daemon osd.6 on smithi169 2023-12-16T07:47:42.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:41 smithi007 bash[64479]: cluster 2023-12-16T07:47:41.141238+0000 mgr.y (mgr.25477) 128 : cluster 0 pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:44.056 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:43 smithi169 bash[51638]: audit 2023-12-16T07:47:42.790093+0000 mon.a (mon.0) 206 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:44.056 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:43 smithi169 bash[51638]: audit 2023-12-16T07:47:42.800419+0000 mon.a (mon.0) 207 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:44.056 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:43 smithi169 bash[51638]: cephadm 2023-12-16T07:47:42.809625+0000 mgr.y (mgr.25477) 129 : cephadm 1 Reconfiguring osd.7 (monmap changed)... 2023-12-16T07:47:44.056 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:43 smithi169 bash[51638]: audit 2023-12-16T07:47:42.810299+0000 mon.a (mon.0) 208 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-12-16T07:47:44.056 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:43 smithi169 bash[51638]: audit 2023-12-16T07:47:42.811841+0000 mon.a (mon.0) 209 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:44.056 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:43 smithi169 bash[51638]: cephadm 2023-12-16T07:47:42.817171+0000 mgr.y (mgr.25477) 130 : cephadm 1 Reconfiguring daemon osd.7 on smithi169 2023-12-16T07:47:44.056 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:43 smithi169 bash[51638]: cluster 2023-12-16T07:47:43.141993+0000 mgr.y (mgr.25477) 131 : cluster 0 pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:44.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:43 smithi007 bash[64479]: audit 2023-12-16T07:47:42.790093+0000 mon.a (mon.0) 206 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:44.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:43 smithi007 bash[64479]: audit 2023-12-16T07:47:42.800419+0000 mon.a (mon.0) 207 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:44.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:43 smithi007 bash[64479]: cephadm 2023-12-16T07:47:42.809625+0000 mgr.y (mgr.25477) 129 : cephadm 1 Reconfiguring osd.7 (monmap changed)... 2023-12-16T07:47:44.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:43 smithi007 bash[64479]: audit 2023-12-16T07:47:42.810299+0000 mon.a (mon.0) 208 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-12-16T07:47:44.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:43 smithi007 bash[64479]: audit 2023-12-16T07:47:42.811841+0000 mon.a (mon.0) 209 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:44.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:43 smithi007 bash[64479]: cephadm 2023-12-16T07:47:42.817171+0000 mgr.y (mgr.25477) 130 : cephadm 1 Reconfiguring daemon osd.7 on smithi169 2023-12-16T07:47:44.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:43 smithi007 bash[64479]: cluster 2023-12-16T07:47:43.141993+0000 mgr.y (mgr.25477) 131 : cluster 0 pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:44.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:43 smithi007 bash[65864]: audit 2023-12-16T07:47:42.790093+0000 mon.a (mon.0) 206 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:44.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:43 smithi007 bash[65864]: audit 2023-12-16T07:47:42.800419+0000 mon.a (mon.0) 207 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:44.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:43 smithi007 bash[65864]: cephadm 2023-12-16T07:47:42.809625+0000 mgr.y (mgr.25477) 129 : cephadm 1 Reconfiguring osd.7 (monmap changed)... 2023-12-16T07:47:44.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:43 smithi007 bash[65864]: audit 2023-12-16T07:47:42.810299+0000 mon.a (mon.0) 208 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-12-16T07:47:44.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:43 smithi007 bash[65864]: audit 2023-12-16T07:47:42.811841+0000 mon.a (mon.0) 209 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:44.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:43 smithi007 bash[65864]: cephadm 2023-12-16T07:47:42.817171+0000 mgr.y (mgr.25477) 130 : cephadm 1 Reconfiguring daemon osd.7 on smithi169 2023-12-16T07:47:44.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:43 smithi007 bash[65864]: cluster 2023-12-16T07:47:43.141993+0000 mgr.y (mgr.25477) 131 : cluster 0 pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:45.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:44 smithi007 bash[65864]: audit 2023-12-16T07:47:44.183213+0000 mgr.y (mgr.25477) 132 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:45.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:44 smithi007 bash[64479]: audit 2023-12-16T07:47:44.183213+0000 mgr.y (mgr.25477) 132 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:45.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:44 smithi169 bash[51638]: audit 2023-12-16T07:47:44.183213+0000 mgr.y (mgr.25477) 132 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:46.231 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:45 smithi169 bash[51638]: audit 2023-12-16T07:47:44.934105+0000 mon.a (mon.0) 210 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:46.231 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:45 smithi169 bash[51638]: audit 2023-12-16T07:47:44.944853+0000 mon.a (mon.0) 211 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:46.231 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:45 smithi169 bash[51638]: cephadm 2023-12-16T07:47:44.987735+0000 mgr.y (mgr.25477) 133 : cephadm 1 Reconfiguring rgw.foo.smithi169.okfkhg (monmap changed)... 2023-12-16T07:47:46.231 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:45 smithi169 bash[51638]: audit 2023-12-16T07:47:44.988627+0000 mon.a (mon.0) 212 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi169.okfkhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:47:46.231 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:45 smithi169 bash[51638]: audit 2023-12-16T07:47:44.993116+0000 mon.a (mon.0) 213 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:46.231 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:45 smithi169 bash[51638]: cephadm 2023-12-16T07:47:44.995338+0000 mgr.y (mgr.25477) 134 : cephadm 1 Reconfiguring daemon rgw.foo.smithi169.okfkhg on smithi169 2023-12-16T07:47:46.231 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:45 smithi169 bash[51638]: cluster 2023-12-16T07:47:45.143688+0000 mgr.y (mgr.25477) 135 : cluster 0 pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:46.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:45 smithi007 bash[64479]: audit 2023-12-16T07:47:44.934105+0000 mon.a (mon.0) 210 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:46.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:45 smithi007 bash[64479]: audit 2023-12-16T07:47:44.944853+0000 mon.a (mon.0) 211 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:46.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:45 smithi007 bash[64479]: cephadm 2023-12-16T07:47:44.987735+0000 mgr.y (mgr.25477) 133 : cephadm 1 Reconfiguring rgw.foo.smithi169.okfkhg (monmap changed)... 2023-12-16T07:47:46.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:45 smithi007 bash[64479]: audit 2023-12-16T07:47:44.988627+0000 mon.a (mon.0) 212 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi169.okfkhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:47:46.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:45 smithi007 bash[64479]: audit 2023-12-16T07:47:44.993116+0000 mon.a (mon.0) 213 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:46.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:45 smithi007 bash[64479]: cephadm 2023-12-16T07:47:44.995338+0000 mgr.y (mgr.25477) 134 : cephadm 1 Reconfiguring daemon rgw.foo.smithi169.okfkhg on smithi169 2023-12-16T07:47:46.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:45 smithi007 bash[64479]: cluster 2023-12-16T07:47:45.143688+0000 mgr.y (mgr.25477) 135 : cluster 0 pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:46.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:45 smithi007 bash[65864]: audit 2023-12-16T07:47:44.934105+0000 mon.a (mon.0) 210 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:46.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:45 smithi007 bash[65864]: audit 2023-12-16T07:47:44.944853+0000 mon.a (mon.0) 211 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:46.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:45 smithi007 bash[65864]: cephadm 2023-12-16T07:47:44.987735+0000 mgr.y (mgr.25477) 133 : cephadm 1 Reconfiguring rgw.foo.smithi169.okfkhg (monmap changed)... 2023-12-16T07:47:46.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:45 smithi007 bash[65864]: audit 2023-12-16T07:47:44.988627+0000 mon.a (mon.0) 212 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi169.okfkhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:47:46.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:45 smithi007 bash[65864]: audit 2023-12-16T07:47:44.993116+0000 mon.a (mon.0) 213 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:46.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:45 smithi007 bash[65864]: cephadm 2023-12-16T07:47:44.995338+0000 mgr.y (mgr.25477) 134 : cephadm 1 Reconfiguring daemon rgw.foo.smithi169.okfkhg on smithi169 2023-12-16T07:47:46.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:45 smithi007 bash[65864]: cluster 2023-12-16T07:47:45.143688+0000 mgr.y (mgr.25477) 135 : cluster 0 pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:48.388 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:48 smithi169 bash[51638]: audit 2023-12-16T07:47:47.075402+0000 mon.a (mon.0) 214 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:48.388 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:48 smithi169 bash[51638]: audit 2023-12-16T07:47:47.083467+0000 mon.a (mon.0) 215 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:48.388 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:48 smithi169 bash[51638]: cephadm 2023-12-16T07:47:47.115247+0000 mgr.y (mgr.25477) 136 : cephadm 1 Reconfiguring rgw.smpl.smithi169.aiyfmr (monmap changed)... 2023-12-16T07:47:48.388 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:48 smithi169 bash[51638]: audit 2023-12-16T07:47:47.116012+0000 mon.a (mon.0) 216 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi169.aiyfmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:47:48.388 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:48 smithi169 bash[51638]: audit 2023-12-16T07:47:47.119666+0000 mon.a (mon.0) 217 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:48.388 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:48 smithi169 bash[51638]: cephadm 2023-12-16T07:47:47.121718+0000 mgr.y (mgr.25477) 137 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi169.aiyfmr on smithi169 2023-12-16T07:47:48.388 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:48 smithi169 bash[51638]: cluster 2023-12-16T07:47:47.144447+0000 mgr.y (mgr.25477) 138 : cluster 0 pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:48.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:48 smithi007 bash[64479]: audit 2023-12-16T07:47:47.075402+0000 mon.a (mon.0) 214 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:48.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:48 smithi007 bash[64479]: audit 2023-12-16T07:47:47.083467+0000 mon.a (mon.0) 215 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:48.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:48 smithi007 bash[64479]: cephadm 2023-12-16T07:47:47.115247+0000 mgr.y (mgr.25477) 136 : cephadm 1 Reconfiguring rgw.smpl.smithi169.aiyfmr (monmap changed)... 2023-12-16T07:47:48.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:48 smithi007 bash[64479]: audit 2023-12-16T07:47:47.116012+0000 mon.a (mon.0) 216 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi169.aiyfmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:47:48.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:48 smithi007 bash[64479]: audit 2023-12-16T07:47:47.119666+0000 mon.a (mon.0) 217 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:48.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:48 smithi007 bash[64479]: cephadm 2023-12-16T07:47:47.121718+0000 mgr.y (mgr.25477) 137 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi169.aiyfmr on smithi169 2023-12-16T07:47:48.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:48 smithi007 bash[64479]: cluster 2023-12-16T07:47:47.144447+0000 mgr.y (mgr.25477) 138 : cluster 0 pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:48.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:48 smithi007 bash[65864]: audit 2023-12-16T07:47:47.075402+0000 mon.a (mon.0) 214 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:48.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:48 smithi007 bash[65864]: audit 2023-12-16T07:47:47.083467+0000 mon.a (mon.0) 215 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:48.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:48 smithi007 bash[65864]: cephadm 2023-12-16T07:47:47.115247+0000 mgr.y (mgr.25477) 136 : cephadm 1 Reconfiguring rgw.smpl.smithi169.aiyfmr (monmap changed)... 2023-12-16T07:47:48.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:48 smithi007 bash[65864]: audit 2023-12-16T07:47:47.116012+0000 mon.a (mon.0) 216 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi169.aiyfmr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-12-16T07:47:48.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:48 smithi007 bash[65864]: audit 2023-12-16T07:47:47.119666+0000 mon.a (mon.0) 217 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:48.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:48 smithi007 bash[65864]: cephadm 2023-12-16T07:47:47.121718+0000 mgr.y (mgr.25477) 137 : cephadm 1 Reconfiguring daemon rgw.smpl.smithi169.aiyfmr on smithi169 2023-12-16T07:47:48.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:48 smithi007 bash[65864]: cluster 2023-12-16T07:47:47.144447+0000 mgr.y (mgr.25477) 138 : cluster 0 pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:48.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:47:48 smithi169 bash[49087]: ts=2023-12-16T07:47:48.662Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:47:50.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: cluster 2023-12-16T07:47:49.146090+0000 mgr.y (mgr.25477) 139 : cluster 0 pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:50.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.170708+0000 mon.a (mon.0) 218 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:50.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.180711+0000 mon.a (mon.0) 219 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:50.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.199415+0000 mon.a (mon.0) 220 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:47:50.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.202455+0000 mon.a (mon.0) 221 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:50.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.205039+0000 mon.a (mon.0) 222 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:50.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: cephadm 2023-12-16T07:47:49.206135+0000 mgr.y (mgr.25477) 140 : cephadm 1 Upgrade: Setting container_image for all mon 2023-12-16T07:47:50.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.212182+0000 mon.a (mon.0) 223 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:50.435 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.215963+0000 mon.a (mon.0) 224 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2023-12-16T07:47:50.435 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.220155+0000 mon.a (mon.0) 225 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2023-12-16T07:47:50.435 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.223099+0000 mon.a (mon.0) 226 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2023-12-16T07:47:50.435 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.228489+0000 mon.a (mon.0) 227 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2023-12-16T07:47:50.435 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.232234+0000 mon.a (mon.0) 228 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2023-12-16T07:47:50.435 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.236915+0000 mon.a (mon.0) 229 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2023-12-16T07:47:50.436 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.240967+0000 mon.a (mon.0) 230 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:50.436 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: cephadm 2023-12-16T07:47:49.242127+0000 mgr.y (mgr.25477) 141 : cephadm 1 Upgrade: Setting container_image for all crash 2023-12-16T07:47:50.436 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.246276+0000 mon.a (mon.0) 231 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:50.436 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.249843+0000 mon.a (mon.0) 232 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-12-16T07:47:50.436 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: audit 2023-12-16T07:47:49.250121+0000 mgr.y (mgr.25477) 142 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-12-16T07:47:50.436 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:50 smithi169 bash[51638]: cephadm 2023-12-16T07:47:49.251816+0000 mgr.y (mgr.25477) 143 : cephadm 1 Upgrade: osd.0 is safe to restart 2023-12-16T07:47:50.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:47:50] "GET /metrics HTTP/1.1" 200 38169 "" "Prometheus/2.33.4" 2023-12-16T07:47:50.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: cluster 2023-12-16T07:47:49.146090+0000 mgr.y (mgr.25477) 139 : cluster 0 pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:50.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.170708+0000 mon.a (mon.0) 218 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:50.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.180711+0000 mon.a (mon.0) 219 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:50.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.199415+0000 mon.a (mon.0) 220 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:47:50.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.202455+0000 mon.a (mon.0) 221 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:50.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.205039+0000 mon.a (mon.0) 222 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:50.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: cephadm 2023-12-16T07:47:49.206135+0000 mgr.y (mgr.25477) 140 : cephadm 1 Upgrade: Setting container_image for all mon 2023-12-16T07:47:50.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.212182+0000 mon.a (mon.0) 223 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:50.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.215963+0000 mon.a (mon.0) 224 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2023-12-16T07:47:50.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.220155+0000 mon.a (mon.0) 225 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2023-12-16T07:47:50.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.223099+0000 mon.a (mon.0) 226 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2023-12-16T07:47:50.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.228489+0000 mon.a (mon.0) 227 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2023-12-16T07:47:50.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.232234+0000 mon.a (mon.0) 228 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2023-12-16T07:47:50.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.236915+0000 mon.a (mon.0) 229 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2023-12-16T07:47:50.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.240967+0000 mon.a (mon.0) 230 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:50.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: cephadm 2023-12-16T07:47:49.242127+0000 mgr.y (mgr.25477) 141 : cephadm 1 Upgrade: Setting container_image for all crash 2023-12-16T07:47:50.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.246276+0000 mon.a (mon.0) 231 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:50.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.249843+0000 mon.a (mon.0) 232 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-12-16T07:47:50.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: audit 2023-12-16T07:47:49.250121+0000 mgr.y (mgr.25477) 142 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-12-16T07:47:50.572 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[64479]: cephadm 2023-12-16T07:47:49.251816+0000 mgr.y (mgr.25477) 143 : cephadm 1 Upgrade: osd.0 is safe to restart 2023-12-16T07:47:50.572 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: cluster 2023-12-16T07:47:49.146090+0000 mgr.y (mgr.25477) 139 : cluster 0 pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:50.572 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.170708+0000 mon.a (mon.0) 218 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:50.572 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.180711+0000 mon.a (mon.0) 219 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:50.572 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.199415+0000 mon.a (mon.0) 220 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:47:50.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.202455+0000 mon.a (mon.0) 221 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:50.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.205039+0000 mon.a (mon.0) 222 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:50.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: cephadm 2023-12-16T07:47:49.206135+0000 mgr.y (mgr.25477) 140 : cephadm 1 Upgrade: Setting container_image for all mon 2023-12-16T07:47:50.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.212182+0000 mon.a (mon.0) 223 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:50.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.215963+0000 mon.a (mon.0) 224 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2023-12-16T07:47:50.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.220155+0000 mon.a (mon.0) 225 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2023-12-16T07:47:50.574 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.223099+0000 mon.a (mon.0) 226 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2023-12-16T07:47:50.574 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.228489+0000 mon.a (mon.0) 227 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2023-12-16T07:47:50.574 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.232234+0000 mon.a (mon.0) 228 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2023-12-16T07:47:50.574 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.236915+0000 mon.a (mon.0) 229 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2023-12-16T07:47:50.574 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.240967+0000 mon.a (mon.0) 230 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:47:50.575 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: cephadm 2023-12-16T07:47:49.242127+0000 mgr.y (mgr.25477) 141 : cephadm 1 Upgrade: Setting container_image for all crash 2023-12-16T07:47:50.575 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.246276+0000 mon.a (mon.0) 231 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:50.575 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.249843+0000 mon.a (mon.0) 232 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-12-16T07:47:50.575 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: audit 2023-12-16T07:47:49.250121+0000 mgr.y (mgr.25477) 142 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2023-12-16T07:47:50.575 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:50 smithi007 bash[65864]: cephadm 2023-12-16T07:47:49.251816+0000 mgr.y (mgr.25477) 143 : cephadm 1 Upgrade: osd.0 is safe to restart 2023-12-16T07:47:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:52 smithi169 bash[51638]: cluster 2023-12-16T07:47:51.146860+0000 mgr.y (mgr.25477) 144 : cluster 0 pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:52 smithi169 bash[51638]: audit 2023-12-16T07:47:51.828482+0000 mon.a (mon.0) 233 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:52 smithi169 bash[51638]: audit 2023-12-16T07:47:51.832479+0000 mon.a (mon.0) 234 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-16T07:47:52.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:52 smithi169 bash[51638]: audit 2023-12-16T07:47:51.833670+0000 mon.a (mon.0) 235 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:52.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:52 smithi007 bash[65864]: cluster 2023-12-16T07:47:51.146860+0000 mgr.y (mgr.25477) 144 : cluster 0 pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:52.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:52 smithi007 bash[65864]: audit 2023-12-16T07:47:51.828482+0000 mon.a (mon.0) 233 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:52.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:52 smithi007 bash[65864]: audit 2023-12-16T07:47:51.832479+0000 mon.a (mon.0) 234 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-16T07:47:52.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:52 smithi007 bash[65864]: audit 2023-12-16T07:47:51.833670+0000 mon.a (mon.0) 235 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:52.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:52 smithi007 bash[64479]: cluster 2023-12-16T07:47:51.146860+0000 mgr.y (mgr.25477) 144 : cluster 0 pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:52.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:52 smithi007 bash[64479]: audit 2023-12-16T07:47:51.828482+0000 mon.a (mon.0) 233 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:47:52.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:52 smithi007 bash[64479]: audit 2023-12-16T07:47:51.832479+0000 mon.a (mon.0) 234 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-12-16T07:47:52.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:52 smithi007 bash[64479]: audit 2023-12-16T07:47:51.833670+0000 mon.a (mon.0) 235 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:47:53.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:53 smithi007 bash[65864]: cephadm 2023-12-16T07:47:51.821680+0000 mgr.y (mgr.25477) 145 : cephadm 1 Upgrade: Updating osd.0 2023-12-16T07:47:53.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:53 smithi007 bash[65864]: cephadm 2023-12-16T07:47:51.838536+0000 mgr.y (mgr.25477) 146 : cephadm 1 Deploying daemon osd.0 on smithi007 2023-12-16T07:47:53.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:53 smithi007 bash[64479]: cephadm 2023-12-16T07:47:51.821680+0000 mgr.y (mgr.25477) 145 : cephadm 1 Upgrade: Updating osd.0 2023-12-16T07:47:53.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:53 smithi007 bash[64479]: cephadm 2023-12-16T07:47:51.838536+0000 mgr.y (mgr.25477) 146 : cephadm 1 Deploying daemon osd.0 on smithi007 2023-12-16T07:47:53.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:53 smithi169 bash[51638]: cephadm 2023-12-16T07:47:51.821680+0000 mgr.y (mgr.25477) 145 : cephadm 1 Upgrade: Updating osd.0 2023-12-16T07:47:53.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:53 smithi169 bash[51638]: cephadm 2023-12-16T07:47:51.838536+0000 mgr.y (mgr.25477) 146 : cephadm 1 Deploying daemon osd.0 on smithi007 2023-12-16T07:47:54.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:54 smithi007 bash[64479]: cluster 2023-12-16T07:47:53.147573+0000 mgr.y (mgr.25477) 147 : cluster 0 pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:54.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:54 smithi007 bash[64479]: audit 2023-12-16T07:47:53.201529+0000 mon.a (mon.0) 236 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:47:54.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:54 smithi007 bash[65864]: cluster 2023-12-16T07:47:53.147573+0000 mgr.y (mgr.25477) 147 : cluster 0 pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:54.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:54 smithi007 bash[65864]: audit 2023-12-16T07:47:53.201529+0000 mon.a (mon.0) 236 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:47:54.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:54 smithi169 bash[51638]: cluster 2023-12-16T07:47:53.147573+0000 mgr.y (mgr.25477) 147 : cluster 0 pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:54.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:54 smithi169 bash[51638]: audit 2023-12-16T07:47:53.201529+0000 mon.a (mon.0) 236 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:47:55.486 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:55 smithi007 bash[65864]: audit 2023-12-16T07:47:54.191317+0000 mgr.y (mgr.25477) 148 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:55.486 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:55 smithi007 bash[64479]: audit 2023-12-16T07:47:54.191317+0000 mgr.y (mgr.25477) 148 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:55 smithi169 bash[51638]: audit 2023-12-16T07:47:54.191317+0000 mgr.y (mgr.25477) 148 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:47:56.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:56 smithi007 bash[65864]: cluster 2023-12-16T07:47:55.149388+0000 mgr.y (mgr.25477) 149 : cluster 0 pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:56.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:56 smithi007 bash[64479]: cluster 2023-12-16T07:47:55.149388+0000 mgr.y (mgr.25477) 149 : cluster 0 pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:56.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:56 smithi169 bash[51638]: cluster 2023-12-16T07:47:55.149388+0000 mgr.y (mgr.25477) 149 : cluster 0 pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:47:56.931 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:47:56 smithi007 systemd[1]: Stopping Ceph osd.0 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:47:57.205 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:47:56 smithi007 bash[33229]: debug 2023-12-16T07:47:56.925+0000 7faeddcb4700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:47:57.205 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:47:56 smithi007 bash[33229]: debug 2023-12-16T07:47:56.925+0000 7faeddcb4700 -1 osd.0 103 *** Got signal Terminated *** 2023-12-16T07:47:57.205 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:47:56 smithi007 bash[33229]: debug 2023-12-16T07:47:56.925+0000 7faeddcb4700 -1 osd.0 103 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:47:57.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:57 smithi007 bash[65864]: cluster 2023-12-16T07:47:56.929988+0000 mon.a (mon.0) 237 : cluster 1 osd.0 marked itself down and dead 2023-12-16T07:47:57.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:57 smithi007 bash[64479]: cluster 2023-12-16T07:47:56.929988+0000 mon.a (mon.0) 237 : cluster 1 osd.0 marked itself down and dead 2023-12-16T07:47:57.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:57 smithi169 bash[51638]: cluster 2023-12-16T07:47:56.929988+0000 mon.a (mon.0) 237 : cluster 1 osd.0 marked itself down and dead 2023-12-16T07:47:58.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[65864]: cluster 2023-12-16T07:47:57.150218+0000 mgr.y (mgr.25477) 150 : cluster 0 pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:58.474 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[65864]: cluster 2023-12-16T07:47:57.195788+0000 mon.a (mon.0) 238 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:47:58.474 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[65864]: cluster 2023-12-16T07:47:57.203348+0000 osd.1 (osd.1) 4 : cluster 3 failed to encode map e104 with expected crc 2023-12-16T07:47:58.474 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[65864]: cluster 2023-12-16T07:47:57.206593+0000 osd.7 (osd.7) 4 : cluster 3 failed to encode map e104 with expected crc 2023-12-16T07:47:58.474 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[65864]: cluster 2023-12-16T07:47:57.207763+0000 mon.a (mon.0) 239 : cluster 0 osdmap e104: 8 total, 7 up, 8 in 2023-12-16T07:47:58.474 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[65864]: cluster 2023-12-16T07:47:57.208813+0000 osd.3 (osd.3) 4 : cluster 3 failed to encode map e104 with expected crc 2023-12-16T07:47:58.474 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[65864]: cluster 2023-12-16T07:47:57.211675+0000 osd.5 (osd.5) 4 : cluster 3 failed to encode map e104 with expected crc 2023-12-16T07:47:58.474 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[68187]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-0 2023-12-16T07:47:58.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[64479]: cluster 2023-12-16T07:47:57.150218+0000 mgr.y (mgr.25477) 150 : cluster 0 pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:58.475 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[64479]: cluster 2023-12-16T07:47:57.195788+0000 mon.a (mon.0) 238 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:47:58.475 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[64479]: cluster 2023-12-16T07:47:57.203348+0000 osd.1 (osd.1) 4 : cluster 3 failed to encode map e104 with expected crc 2023-12-16T07:47:58.475 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[64479]: cluster 2023-12-16T07:47:57.206593+0000 osd.7 (osd.7) 4 : cluster 3 failed to encode map e104 with expected crc 2023-12-16T07:47:58.475 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[64479]: cluster 2023-12-16T07:47:57.207763+0000 mon.a (mon.0) 239 : cluster 0 osdmap e104: 8 total, 7 up, 8 in 2023-12-16T07:47:58.475 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[64479]: cluster 2023-12-16T07:47:57.208813+0000 osd.3 (osd.3) 4 : cluster 3 failed to encode map e104 with expected crc 2023-12-16T07:47:58.475 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[64479]: cluster 2023-12-16T07:47:57.211675+0000 osd.5 (osd.5) 4 : cluster 3 failed to encode map e104 with expected crc 2023-12-16T07:47:58.663 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:58 smithi169 bash[51638]: cluster 2023-12-16T07:47:57.150218+0000 mgr.y (mgr.25477) 150 : cluster 0 pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:47:58.663 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:58 smithi169 bash[51638]: cluster 2023-12-16T07:47:57.195788+0000 mon.a (mon.0) 238 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:47:58.663 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:58 smithi169 bash[51638]: cluster 2023-12-16T07:47:57.203348+0000 osd.1 (osd.1) 4 : cluster 3 failed to encode map e104 with expected crc 2023-12-16T07:47:58.663 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:58 smithi169 bash[51638]: cluster 2023-12-16T07:47:57.206593+0000 osd.7 (osd.7) 4 : cluster 3 failed to encode map e104 with expected crc 2023-12-16T07:47:58.663 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:58 smithi169 bash[51638]: cluster 2023-12-16T07:47:57.207763+0000 mon.a (mon.0) 239 : cluster 0 osdmap e104: 8 total, 7 up, 8 in 2023-12-16T07:47:58.663 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:58 smithi169 bash[51638]: cluster 2023-12-16T07:47:57.208813+0000 osd.3 (osd.3) 4 : cluster 3 failed to encode map e104 with expected crc 2023-12-16T07:47:58.663 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:58 smithi169 bash[51638]: cluster 2023-12-16T07:47:57.211675+0000 osd.5 (osd.5) 4 : cluster 3 failed to encode map e104 with expected crc 2023-12-16T07:47:58.817 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:47:58 smithi007 bash[68260]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-0 2023-12-16T07:47:58.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:47:58 smithi169 bash[49087]: ts=2023-12-16T07:47:58.662Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:47:59.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[65864]: cluster 2023-12-16T07:47:58.215873+0000 osd.6 (osd.6) 4 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[65864]: cluster 2023-12-16T07:47:58.216066+0000 osd.7 (osd.7) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[65864]: cluster 2023-12-16T07:47:58.217763+0000 mon.a (mon.0) 240 : cluster 0 osdmap e105: 8 total, 7 up, 8 in 2023-12-16T07:47:59.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[65864]: cluster 2023-12-16T07:47:58.218349+0000 osd.3 (osd.3) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[65864]: cluster 2023-12-16T07:47:58.218426+0000 osd.1 (osd.1) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[65864]: cluster 2023-12-16T07:47:58.218531+0000 osd.2 (osd.2) 4 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[65864]: cluster 2023-12-16T07:47:58.220023+0000 osd.6 (osd.6) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[65864]: cluster 2023-12-16T07:47:58.220490+0000 osd.4 (osd.4) 4 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[65864]: cluster 2023-12-16T07:47:58.220734+0000 osd.5 (osd.5) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[64479]: cluster 2023-12-16T07:47:58.215873+0000 osd.6 (osd.6) 4 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[64479]: cluster 2023-12-16T07:47:58.216066+0000 osd.7 (osd.7) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[64479]: cluster 2023-12-16T07:47:58.217763+0000 mon.a (mon.0) 240 : cluster 0 osdmap e105: 8 total, 7 up, 8 in 2023-12-16T07:47:59.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[64479]: cluster 2023-12-16T07:47:58.218349+0000 osd.3 (osd.3) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[64479]: cluster 2023-12-16T07:47:58.218426+0000 osd.1 (osd.1) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[64479]: cluster 2023-12-16T07:47:58.218531+0000 osd.2 (osd.2) 4 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[64479]: cluster 2023-12-16T07:47:58.220023+0000 osd.6 (osd.6) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[64479]: cluster 2023-12-16T07:47:58.220490+0000 osd.4 (osd.4) 4 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:47:59 smithi007 bash[64479]: cluster 2023-12-16T07:47:58.220734+0000 osd.5 (osd.5) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:59 smithi169 bash[51638]: cluster 2023-12-16T07:47:58.215873+0000 osd.6 (osd.6) 4 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:59 smithi169 bash[51638]: cluster 2023-12-16T07:47:58.216066+0000 osd.7 (osd.7) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:59 smithi169 bash[51638]: cluster 2023-12-16T07:47:58.217763+0000 mon.a (mon.0) 240 : cluster 0 osdmap e105: 8 total, 7 up, 8 in 2023-12-16T07:47:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:59 smithi169 bash[51638]: cluster 2023-12-16T07:47:58.218349+0000 osd.3 (osd.3) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:59 smithi169 bash[51638]: cluster 2023-12-16T07:47:58.218426+0000 osd.1 (osd.1) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:59 smithi169 bash[51638]: cluster 2023-12-16T07:47:58.218531+0000 osd.2 (osd.2) 4 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:59 smithi169 bash[51638]: cluster 2023-12-16T07:47:58.220023+0000 osd.6 (osd.6) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:59 smithi169 bash[51638]: cluster 2023-12-16T07:47:58.220490+0000 osd.4 (osd.4) 4 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:47:59.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:47:59 smithi169 bash[51638]: cluster 2023-12-16T07:47:58.220734+0000 osd.5 (osd.5) 5 : cluster 3 failed to encode map e105 with expected crc 2023-12-16T07:48:00.317 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:48:00 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:48:00] "GET /metrics HTTP/1.1" 200 38166 "" "Prometheus/2.33.4" 2023-12-16T07:48:00.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:00 smithi007 bash[64479]: cluster 2023-12-16T07:47:59.152145+0000 mgr.y (mgr.25477) 151 : cluster 0 pgmap v68: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:00.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:00 smithi007 bash[64479]: cluster 2023-12-16T07:47:59.811690+0000 mon.a (mon.0) 241 : cluster 3 Health check failed: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded (PG_DEGRADED) 2023-12-16T07:48:00.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:00 smithi007 bash[65864]: cluster 2023-12-16T07:47:59.152145+0000 mgr.y (mgr.25477) 151 : cluster 0 pgmap v68: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:00.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:00 smithi007 bash[65864]: cluster 2023-12-16T07:47:59.811690+0000 mon.a (mon.0) 241 : cluster 3 Health check failed: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded (PG_DEGRADED) 2023-12-16T07:48:00.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:00 smithi169 bash[51638]: cluster 2023-12-16T07:47:59.152145+0000 mgr.y (mgr.25477) 151 : cluster 0 pgmap v68: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:00.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:00 smithi169 bash[51638]: cluster 2023-12-16T07:47:59.811690+0000 mon.a (mon.0) 241 : cluster 3 Health check failed: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded (PG_DEGRADED) 2023-12-16T07:48:01.170 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:00 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.0.service: Succeeded. 2023-12-16T07:48:01.170 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:00 smithi007 systemd[1]: Stopped Ceph osd.0 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:48:01.170 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:01 smithi007 systemd[1]: Started Ceph osd.0 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:48:02.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:02 smithi169 bash[51638]: audit 2023-12-16T07:48:01.146999+0000 mon.a (mon.0) 242 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:02.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:02 smithi169 bash[51638]: cluster 2023-12-16T07:48:01.152884+0000 mgr.y (mgr.25477) 152 : cluster 0 pgmap v69: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:02.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:02 smithi169 bash[51638]: audit 2023-12-16T07:48:01.154911+0000 mon.a (mon.0) 243 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:02.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:02 smithi007 bash[65864]: audit 2023-12-16T07:48:01.146999+0000 mon.a (mon.0) 242 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:02.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:02 smithi007 bash[65864]: cluster 2023-12-16T07:48:01.152884+0000 mgr.y (mgr.25477) 152 : cluster 0 pgmap v69: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:02.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:02 smithi007 bash[65864]: audit 2023-12-16T07:48:01.154911+0000 mon.a (mon.0) 243 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:02.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:02 smithi007 bash[64479]: audit 2023-12-16T07:48:01.146999+0000 mon.a (mon.0) 242 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:02.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:02 smithi007 bash[64479]: cluster 2023-12-16T07:48:01.152884+0000 mgr.y (mgr.25477) 152 : cluster 0 pgmap v69: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:02.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:02 smithi007 bash[64479]: audit 2023-12-16T07:48:01.154911+0000 mon.a (mon.0) 243 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:03.067 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:02 smithi007 bash[68430]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-16T07:48:03.067 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:02 smithi007 bash[68430]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2023-12-16T07:48:03.067 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:02 smithi007 bash[68430]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2023-12-16T07:48:03.067 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:02 smithi007 bash[68430]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-12-16T07:48:03.067 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:02 smithi007 bash[68430]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2023-12-16T07:48:03.068 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:02 smithi007 bash[68430]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-16T07:48:03.068 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:02 smithi007 bash[68430]: --> ceph-volume raw activate successful for osd ID: 0 2023-12-16T07:48:04.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:04 smithi007 bash[64479]: cluster 2023-12-16T07:48:03.153852+0000 mgr.y (mgr.25477) 153 : cluster 0 pgmap v70: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:04.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:04 smithi007 bash[65864]: cluster 2023-12-16T07:48:03.153852+0000 mgr.y (mgr.25477) 153 : cluster 0 pgmap v70: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:04.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:04 smithi169 bash[51638]: cluster 2023-12-16T07:48:03.153852+0000 mgr.y (mgr.25477) 153 : cluster 0 pgmap v70: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:05.067 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:04 smithi007 bash[68738]: debug 2023-12-16T07:48:04.677+0000 7fd6975a1700 1 -- 172.21.15.7:0/3547555243 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a0abb036c0 con 0x55a0abb88c00 2023-12-16T07:48:05.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:05 smithi007 bash[64479]: audit 2023-12-16T07:48:04.199251+0000 mgr.y (mgr.25477) 154 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:05.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:05 smithi007 bash[65864]: audit 2023-12-16T07:48:04.199251+0000 mgr.y (mgr.25477) 154 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:05.568 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:05 smithi007 bash[68738]: debug 2023-12-16T07:48:05.221+0000 7fd6a0727640 -1 Falling back to public interface 2023-12-16T07:48:05.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:05 smithi169 bash[51638]: audit 2023-12-16T07:48:04.199251+0000 mgr.y (mgr.25477) 154 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:06.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:06 smithi007 bash[64479]: cluster 2023-12-16T07:48:05.156066+0000 mgr.y (mgr.25477) 155 : cluster 0 pgmap v71: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:06.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:06 smithi007 bash[65864]: cluster 2023-12-16T07:48:05.156066+0000 mgr.y (mgr.25477) 155 : cluster 0 pgmap v71: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:06.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:06 smithi169 bash[51638]: cluster 2023-12-16T07:48:05.156066+0000 mgr.y (mgr.25477) 155 : cluster 0 pgmap v71: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:08.213 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:48:08.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:08 smithi007 bash[64479]: cluster 2023-12-16T07:48:07.157182+0000 mgr.y (mgr.25477) 156 : cluster 0 pgmap v72: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:08.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:08 smithi007 bash[64479]: audit 2023-12-16T07:48:08.207269+0000 mon.a (mon.0) 244 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:08.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:08 smithi007 bash[64479]: audit 2023-12-16T07:48:08.208557+0000 mon.a (mon.0) 245 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:48:08.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:08 smithi007 bash[65864]: cluster 2023-12-16T07:48:07.157182+0000 mgr.y (mgr.25477) 156 : cluster 0 pgmap v72: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:08.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:08 smithi007 bash[65864]: audit 2023-12-16T07:48:08.207269+0000 mon.a (mon.0) 244 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:08.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:08 smithi007 bash[65864]: audit 2023-12-16T07:48:08.208557+0000 mon.a (mon.0) 245 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:48:08.568 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:08 smithi007 bash[68738]: debug 2023-12-16T07:48:08.237+0000 7fd6a0727640 -1 osd.0 0 read_superblock omap replica is missing. 2023-12-16T07:48:08.568 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:08 smithi007 bash[68738]: debug 2023-12-16T07:48:08.277+0000 7fd6a0727640 -1 osd.0 103 log_to_monitors true 2023-12-16T07:48:08.662 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:08 smithi169 bash[51638]: cluster 2023-12-16T07:48:07.157182+0000 mgr.y (mgr.25477) 156 : cluster 0 pgmap v72: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:08.662 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:08 smithi169 bash[51638]: audit 2023-12-16T07:48:08.207269+0000 mon.a (mon.0) 244 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:08.662 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:08 smithi169 bash[51638]: audit 2023-12-16T07:48:08.208557+0000 mon.a (mon.0) 245 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:48:08.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:48:08 smithi169 bash[49087]: ts=2023-12-16T07:48:08.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:48:09.033 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:48:09.033 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (4m) 0s ago 11m 15.5M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:48:09.033 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (3m) 70s ago 11m 43.6M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:48:09.033 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (3m) 0s ago 9m 47.0M - 3.5 e1d6a67b021e 8d8009d0c83f 2023-12-16T07:48:09.033 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283,8765 running (2m) 70s ago 17m 433M - 19.0.0-191-g6d933094 a29c258b5d6f 5295ba35c2cd 2023-12-16T07:48:09.033 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (3m) 0s ago 19m 487M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:48:09.033 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (2m) 0s ago 19m 48.3M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 5a266b45f3ce 2023-12-16T07:48:09.033 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (84s) 70s ago 18m 23.6M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1fc9bf864dda 2023-12-16T07:48:09.033 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (110s) 0s ago 18m 40.3M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1dc91c31fc27 2023-12-16T07:48:09.033 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (4m) 0s ago 11m 11.8M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:48:09.034 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (4m) 70s ago 11m 11.3M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:48:09.034 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (5s) 0s ago 17m - 2782M 19.0.0-191-g6d933094 a29c258b5d6f a00724c6d077 2023-12-16T07:48:09.034 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (16m) 0s ago 16m 60.3M 2782M 17.2.0 e1d6a67b021e 14637e0f389c 2023-12-16T07:48:09.034 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (15m) 0s ago 16m 58.7M 2782M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:48:09.034 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (15m) 0s ago 15m 58.6M 2782M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:48:09.034 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (14m) 70s ago 14m 58.5M 3037M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:48:09.034 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (14m) 70s ago 14m 55.3M 3037M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:48:09.034 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (13m) 70s ago 13m 55.8M 3037M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:48:09.034 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (12m) 70s ago 12m 56.9M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:48:09.034 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (2m) 70s ago 12m 59.0M - 2.33.4 514e6a882f6e 2652ce1b8540 2023-12-16T07:48:09.034 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (9m) 0s ago 9m 87.5M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:48:09.034 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (9m) 70s ago 9m 87.4M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:48:09.035 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (9m) 0s ago 9m 87.3M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:48:09.035 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (9m) 70s ago 9m 87.0M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:48:09.553 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:48:10.038 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:48:10.038 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 3 2023-12-16T07:48:10.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:48:10.038 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:48:10.038 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:48:10.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:48:10.038 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:48:10.038 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7 2023-12-16T07:48:10.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:48:10.038 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:48:10.038 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:48:10.038 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:48:10.039 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:48:10.039 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2023-12-16T07:48:10.039 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 5 2023-12-16T07:48:10.039 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:48:10.039 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:48:10.040 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:09 smithi169 bash[51638]: audit 2023-12-16T07:48:08.165889+0000 mgr.y (mgr.25477) 157 : audit 0 from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:10.040 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:09 smithi169 bash[51638]: audit 2023-12-16T07:48:08.287775+0000 mon.c (mon.1) 7 : audit 1 from='osd.0 [v2:172.21.15.7:6802/2367508686,v1:172.21.15.7:6803/2367508686]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-16T07:48:10.040 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:09 smithi169 bash[51638]: audit 2023-12-16T07:48:08.288227+0000 mon.a (mon.0) 246 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-16T07:48:10.040 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:09 smithi169 bash[51638]: audit 2023-12-16T07:48:08.680577+0000 mon.a (mon.0) 247 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:10.040 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:09 smithi169 bash[51638]: audit 2023-12-16T07:48:08.688674+0000 mon.a (mon.0) 248 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:10.040 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:48:09 smithi007 bash[68738]: debug 2023-12-16T07:48:09.309+0000 7fd690395700 -1 osd.0 103 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:48:10.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:09 smithi007 bash[64479]: audit 2023-12-16T07:48:08.165889+0000 mgr.y (mgr.25477) 157 : audit 0 from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:10.040 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:09 smithi007 bash[64479]: audit 2023-12-16T07:48:08.287775+0000 mon.c (mon.1) 7 : audit 1 from='osd.0 [v2:172.21.15.7:6802/2367508686,v1:172.21.15.7:6803/2367508686]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-16T07:48:10.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:09 smithi007 bash[64479]: audit 2023-12-16T07:48:08.288227+0000 mon.a (mon.0) 246 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-16T07:48:10.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:09 smithi007 bash[64479]: audit 2023-12-16T07:48:08.680577+0000 mon.a (mon.0) 247 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:10.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:09 smithi007 bash[64479]: audit 2023-12-16T07:48:08.688674+0000 mon.a (mon.0) 248 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:10.041 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:09 smithi007 bash[65864]: audit 2023-12-16T07:48:08.165889+0000 mgr.y (mgr.25477) 157 : audit 0 from='client.34278 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:10.041 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:09 smithi007 bash[65864]: audit 2023-12-16T07:48:08.287775+0000 mon.c (mon.1) 7 : audit 1 from='osd.0 [v2:172.21.15.7:6802/2367508686,v1:172.21.15.7:6803/2367508686]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-16T07:48:10.041 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:09 smithi007 bash[65864]: audit 2023-12-16T07:48:08.288227+0000 mon.a (mon.0) 246 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-12-16T07:48:10.041 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:09 smithi007 bash[65864]: audit 2023-12-16T07:48:08.680577+0000 mon.a (mon.0) 247 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:10.041 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:09 smithi007 bash[65864]: audit 2023-12-16T07:48:08.688674+0000 mon.a (mon.0) 248 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:10.042 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:48:10.042 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:48:10.042 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:48:10.042 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:48:10.042 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2023-12-16T07:48:10.042 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr", 2023-12-16T07:48:10.042 INFO:teuthology.orchestra.run.smithi007.stdout: "mon" 2023-12-16T07:48:10.042 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:48:10.042 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "6/23 daemons upgraded", 2023-12-16T07:48:10.042 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading osd daemons", 2023-12-16T07:48:10.042 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:48:10.042 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:48:10.479 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_WARN Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded 2023-12-16T07:48:10.479 INFO:teuthology.orchestra.run.smithi007.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded 2023-12-16T07:48:10.479 INFO:teuthology.orchestra.run.smithi007.stdout: pg 1.0 is active+undersized+degraded, acting [7,6] 2023-12-16T07:48:10.479 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.1 is active+undersized+degraded, acting [2,3] 2023-12-16T07:48:10.479 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.c is active+undersized+degraded, acting [2,5] 2023-12-16T07:48:10.479 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.f is active+undersized+degraded, acting [4,7] 2023-12-16T07:48:10.479 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.13 is active+undersized+degraded, acting [4,3] 2023-12-16T07:48:10.480 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.1e is active+undersized+degraded, acting [3,5] 2023-12-16T07:48:10.480 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.1f is active+undersized+degraded, acting [7,4] 2023-12-16T07:48:10.480 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2023-12-16T07:48:10.480 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.3 is active+undersized+degraded, acting [4,6] 2023-12-16T07:48:10.480 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2023-12-16T07:48:10.480 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.7 is active+undersized+degraded, acting [3,7] 2023-12-16T07:48:10.480 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.b is active+undersized+degraded, acting [3,4] 2023-12-16T07:48:10.480 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.f is active+undersized+degraded, acting [7,4] 2023-12-16T07:48:10.480 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.10 is active+undersized+degraded, acting [6,5] 2023-12-16T07:48:10.481 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2023-12-16T07:48:10.481 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2023-12-16T07:48:10.481 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.18 is active+undersized+degraded, acting [3,1] 2023-12-16T07:48:10.481 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2023-12-16T07:48:10.481 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2023-12-16T07:48:10.481 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.0 is active+undersized+degraded, acting [3,7] 2023-12-16T07:48:10.481 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.3 is active+undersized+degraded, acting [5,7] 2023-12-16T07:48:10.482 INFO:teuthology.orchestra.run.smithi007.stdout: pg 5.1d is active+undersized+degraded, acting [1,4] 2023-12-16T07:48:10.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:48:10] "GET /metrics HTTP/1.1" 200 38316 "" "Prometheus/2.33.4" 2023-12-16T07:48:10.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[64479]: audit 2023-12-16T07:48:08.604848+0000 mgr.y (mgr.25477) 158 : audit 0 from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:10.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[64479]: audit 2023-12-16T07:48:09.012289+0000 mgr.y (mgr.25477) 159 : audit 0 from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:10.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[64479]: cluster 2023-12-16T07:48:09.158626+0000 mgr.y (mgr.25477) 160 : cluster 0 pgmap v73: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:10.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[64479]: audit 2023-12-16T07:48:09.237603+0000 mon.a (mon.0) 249 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-16T07:48:10.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[64479]: cluster 2023-12-16T07:48:09.239043+0000 osd.2 (osd.2) 5 : cluster 3 failed to encode map e106 with expected crc 2023-12-16T07:48:10.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[64479]: cluster 2023-12-16T07:48:09.241292+0000 mon.a (mon.0) 250 : cluster 0 osdmap e106: 8 total, 7 up, 8 in 2023-12-16T07:48:10.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[64479]: cluster 2023-12-16T07:48:09.241586+0000 osd.3 (osd.3) 6 : cluster 3 failed to encode map e106 with expected crc 2023-12-16T07:48:10.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[64479]: audit 2023-12-16T07:48:09.243178+0000 mon.c (mon.1) 8 : audit 1 from='osd.0 [v2:172.21.15.7:6802/2367508686,v1:172.21.15.7:6803/2367508686]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:48:10.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[64479]: audit 2023-12-16T07:48:09.243840+0000 mon.a (mon.0) 251 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:48:10.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[64479]: audit 2023-12-16T07:48:09.547348+0000 mon.a (mon.0) 252 : audit 0 from='client.? 172.21.15.7:0/2477462848' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:10.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[64479]: audit 2023-12-16T07:48:09.736936+0000 mon.a (mon.0) 253 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:10.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[64479]: audit 2023-12-16T07:48:09.745357+0000 mon.a (mon.0) 254 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:10.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[64479]: cluster 2023-12-16T07:48:10.038251+0000 osd.7 (osd.7) 6 : cluster 3 failed to encode map e106 with expected crc 2023-12-16T07:48:10.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[64479]: cluster 2023-12-16T07:48:10.064827+0000 osd.5 (osd.5) 6 : cluster 3 failed to encode map e106 with expected crc 2023-12-16T07:48:10.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[65864]: audit 2023-12-16T07:48:08.604848+0000 mgr.y (mgr.25477) 158 : audit 0 from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:10.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[65864]: audit 2023-12-16T07:48:09.012289+0000 mgr.y (mgr.25477) 159 : audit 0 from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:10.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[65864]: cluster 2023-12-16T07:48:09.158626+0000 mgr.y (mgr.25477) 160 : cluster 0 pgmap v73: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:10.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[65864]: audit 2023-12-16T07:48:09.237603+0000 mon.a (mon.0) 249 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-16T07:48:10.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[65864]: cluster 2023-12-16T07:48:09.239043+0000 osd.2 (osd.2) 5 : cluster 3 failed to encode map e106 with expected crc 2023-12-16T07:48:10.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[65864]: cluster 2023-12-16T07:48:09.241292+0000 mon.a (mon.0) 250 : cluster 0 osdmap e106: 8 total, 7 up, 8 in 2023-12-16T07:48:10.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[65864]: cluster 2023-12-16T07:48:09.241586+0000 osd.3 (osd.3) 6 : cluster 3 failed to encode map e106 with expected crc 2023-12-16T07:48:10.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[65864]: audit 2023-12-16T07:48:09.243178+0000 mon.c (mon.1) 8 : audit 1 from='osd.0 [v2:172.21.15.7:6802/2367508686,v1:172.21.15.7:6803/2367508686]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:48:10.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[65864]: audit 2023-12-16T07:48:09.243840+0000 mon.a (mon.0) 251 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:48:10.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[65864]: audit 2023-12-16T07:48:09.547348+0000 mon.a (mon.0) 252 : audit 0 from='client.? 172.21.15.7:0/2477462848' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:10.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[65864]: audit 2023-12-16T07:48:09.736936+0000 mon.a (mon.0) 253 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:10.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[65864]: audit 2023-12-16T07:48:09.745357+0000 mon.a (mon.0) 254 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:10.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[65864]: cluster 2023-12-16T07:48:10.038251+0000 osd.7 (osd.7) 6 : cluster 3 failed to encode map e106 with expected crc 2023-12-16T07:48:10.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:10 smithi007 bash[65864]: cluster 2023-12-16T07:48:10.064827+0000 osd.5 (osd.5) 6 : cluster 3 failed to encode map e106 with expected crc 2023-12-16T07:48:10.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:10 smithi169 bash[51638]: audit 2023-12-16T07:48:08.604848+0000 mgr.y (mgr.25477) 158 : audit 0 from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:10.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:10 smithi169 bash[51638]: audit 2023-12-16T07:48:09.012289+0000 mgr.y (mgr.25477) 159 : audit 0 from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:10.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:10 smithi169 bash[51638]: cluster 2023-12-16T07:48:09.158626+0000 mgr.y (mgr.25477) 160 : cluster 0 pgmap v73: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 69/630 objects degraded (10.952%) 2023-12-16T07:48:10.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:10 smithi169 bash[51638]: audit 2023-12-16T07:48:09.237603+0000 mon.a (mon.0) 249 : audit 1 from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-12-16T07:48:10.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:10 smithi169 bash[51638]: cluster 2023-12-16T07:48:09.239043+0000 osd.2 (osd.2) 5 : cluster 3 failed to encode map e106 with expected crc 2023-12-16T07:48:10.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:10 smithi169 bash[51638]: cluster 2023-12-16T07:48:09.241292+0000 mon.a (mon.0) 250 : cluster 0 osdmap e106: 8 total, 7 up, 8 in 2023-12-16T07:48:10.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:10 smithi169 bash[51638]: cluster 2023-12-16T07:48:09.241586+0000 osd.3 (osd.3) 6 : cluster 3 failed to encode map e106 with expected crc 2023-12-16T07:48:10.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:10 smithi169 bash[51638]: audit 2023-12-16T07:48:09.243178+0000 mon.c (mon.1) 8 : audit 1 from='osd.0 [v2:172.21.15.7:6802/2367508686,v1:172.21.15.7:6803/2367508686]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:48:10.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:10 smithi169 bash[51638]: audit 2023-12-16T07:48:09.243840+0000 mon.a (mon.0) 251 : audit 1 from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:48:10.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:10 smithi169 bash[51638]: audit 2023-12-16T07:48:09.547348+0000 mon.a (mon.0) 252 : audit 0 from='client.? 172.21.15.7:0/2477462848' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:10.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:10 smithi169 bash[51638]: audit 2023-12-16T07:48:09.736936+0000 mon.a (mon.0) 253 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:10.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:10 smithi169 bash[51638]: audit 2023-12-16T07:48:09.745357+0000 mon.a (mon.0) 254 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:10.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:10 smithi169 bash[51638]: cluster 2023-12-16T07:48:10.038251+0000 osd.7 (osd.7) 6 : cluster 3 failed to encode map e106 with expected crc 2023-12-16T07:48:10.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:10 smithi169 bash[51638]: cluster 2023-12-16T07:48:10.064827+0000 osd.5 (osd.5) 6 : cluster 3 failed to encode map e106 with expected crc 2023-12-16T07:48:11.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[64479]: audit 2023-12-16T07:48:09.943960+0000 mgr.y (mgr.25477) 161 : audit 0 from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:11.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[64479]: cluster 2023-12-16T07:48:10.238305+0000 mon.a (mon.0) 255 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:48:11.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[64479]: cluster 2023-12-16T07:48:10.243848+0000 osd.6 (osd.6) 6 : cluster 3 failed to encode map e107 with expected crc 2023-12-16T07:48:11.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[64479]: cluster 2023-12-16T07:48:10.244605+0000 osd.4 (osd.4) 5 : cluster 3 failed to encode map e107 with expected crc 2023-12-16T07:48:11.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[64479]: cluster 2023-12-16T07:48:10.246299+0000 mon.a (mon.0) 256 : cluster 1 osd.0 [v2:172.21.15.7:6802/2367508686,v1:172.21.15.7:6803/2367508686] boot 2023-12-16T07:48:11.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[64479]: cluster 2023-12-16T07:48:10.246434+0000 mon.a (mon.0) 257 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2023-12-16T07:48:11.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[64479]: audit 2023-12-16T07:48:10.247176+0000 mon.a (mon.0) 258 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:48:11.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[64479]: audit 2023-12-16T07:48:10.475826+0000 mon.a (mon.0) 259 : audit 0 from='client.? 172.21.15.7:0/903647441' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:48:11.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[65864]: audit 2023-12-16T07:48:09.943960+0000 mgr.y (mgr.25477) 161 : audit 0 from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:11.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[65864]: cluster 2023-12-16T07:48:10.238305+0000 mon.a (mon.0) 255 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:48:11.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[65864]: cluster 2023-12-16T07:48:10.243848+0000 osd.6 (osd.6) 6 : cluster 3 failed to encode map e107 with expected crc 2023-12-16T07:48:11.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[65864]: cluster 2023-12-16T07:48:10.244605+0000 osd.4 (osd.4) 5 : cluster 3 failed to encode map e107 with expected crc 2023-12-16T07:48:11.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[65864]: cluster 2023-12-16T07:48:10.246299+0000 mon.a (mon.0) 256 : cluster 1 osd.0 [v2:172.21.15.7:6802/2367508686,v1:172.21.15.7:6803/2367508686] boot 2023-12-16T07:48:11.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[65864]: cluster 2023-12-16T07:48:10.246434+0000 mon.a (mon.0) 257 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2023-12-16T07:48:11.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[65864]: audit 2023-12-16T07:48:10.247176+0000 mon.a (mon.0) 258 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:48:11.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:11 smithi007 bash[65864]: audit 2023-12-16T07:48:10.475826+0000 mon.a (mon.0) 259 : audit 0 from='client.? 172.21.15.7:0/903647441' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:48:11.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:11 smithi169 bash[51638]: audit 2023-12-16T07:48:09.943960+0000 mgr.y (mgr.25477) 161 : audit 0 from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:11.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:11 smithi169 bash[51638]: cluster 2023-12-16T07:48:10.238305+0000 mon.a (mon.0) 255 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:48:11.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:11 smithi169 bash[51638]: cluster 2023-12-16T07:48:10.243848+0000 osd.6 (osd.6) 6 : cluster 3 failed to encode map e107 with expected crc 2023-12-16T07:48:11.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:11 smithi169 bash[51638]: cluster 2023-12-16T07:48:10.244605+0000 osd.4 (osd.4) 5 : cluster 3 failed to encode map e107 with expected crc 2023-12-16T07:48:11.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:11 smithi169 bash[51638]: cluster 2023-12-16T07:48:10.246299+0000 mon.a (mon.0) 256 : cluster 1 osd.0 [v2:172.21.15.7:6802/2367508686,v1:172.21.15.7:6803/2367508686] boot 2023-12-16T07:48:11.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:11 smithi169 bash[51638]: cluster 2023-12-16T07:48:10.246434+0000 mon.a (mon.0) 257 : cluster 0 osdmap e107: 8 total, 8 up, 8 in 2023-12-16T07:48:11.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:11 smithi169 bash[51638]: audit 2023-12-16T07:48:10.247176+0000 mon.a (mon.0) 258 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-12-16T07:48:11.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:11 smithi169 bash[51638]: audit 2023-12-16T07:48:10.475826+0000 mon.a (mon.0) 259 : audit 0 from='client.? 172.21.15.7:0/903647441' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:48:12.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[64479]: cluster 2023-12-16T07:48:11.159819+0000 mgr.y (mgr.25477) 162 : cluster 0 pgmap v76: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 69/630 objects degraded (10.952%) 2023-12-16T07:48:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[64479]: cluster 2023-12-16T07:48:11.261456+0000 osd.1 (osd.1) 6 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[64479]: cluster 2023-12-16T07:48:11.261998+0000 osd.2 (osd.2) 6 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[64479]: cluster 2023-12-16T07:48:11.264754+0000 mon.a (mon.0) 260 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2023-12-16T07:48:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[64479]: cluster 2023-12-16T07:48:11.264918+0000 osd.1 (osd.1) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[64479]: cluster 2023-12-16T07:48:11.266156+0000 osd.3 (osd.3) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[64479]: cluster 2023-12-16T07:48:11.266389+0000 osd.2 (osd.2) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[64479]: cluster 2023-12-16T07:48:11.267045+0000 osd.6 (osd.6) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[64479]: cluster 2023-12-16T07:48:11.268376+0000 osd.7 (osd.7) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[64479]: cluster 2023-12-16T07:48:11.268477+0000 osd.5 (osd.5) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[64479]: cluster 2023-12-16T07:48:11.268519+0000 osd.4 (osd.4) 6 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[64479]: cluster 2023-12-16T07:48:11.268774+0000 osd.2 (osd.2) 8 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[65864]: cluster 2023-12-16T07:48:11.159819+0000 mgr.y (mgr.25477) 162 : cluster 0 pgmap v76: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 69/630 objects degraded (10.952%) 2023-12-16T07:48:12.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[65864]: cluster 2023-12-16T07:48:11.261456+0000 osd.1 (osd.1) 6 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[65864]: cluster 2023-12-16T07:48:11.261998+0000 osd.2 (osd.2) 6 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[65864]: cluster 2023-12-16T07:48:11.264754+0000 mon.a (mon.0) 260 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2023-12-16T07:48:12.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[65864]: cluster 2023-12-16T07:48:11.264918+0000 osd.1 (osd.1) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[65864]: cluster 2023-12-16T07:48:11.266156+0000 osd.3 (osd.3) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[65864]: cluster 2023-12-16T07:48:11.266389+0000 osd.2 (osd.2) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[65864]: cluster 2023-12-16T07:48:11.267045+0000 osd.6 (osd.6) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[65864]: cluster 2023-12-16T07:48:11.268376+0000 osd.7 (osd.7) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[65864]: cluster 2023-12-16T07:48:11.268477+0000 osd.5 (osd.5) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[65864]: cluster 2023-12-16T07:48:11.268519+0000 osd.4 (osd.4) 6 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:12 smithi007 bash[65864]: cluster 2023-12-16T07:48:11.268774+0000 osd.2 (osd.2) 8 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:12 smithi169 bash[51638]: cluster 2023-12-16T07:48:11.159819+0000 mgr.y (mgr.25477) 162 : cluster 0 pgmap v76: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 69/630 objects degraded (10.952%) 2023-12-16T07:48:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:12 smithi169 bash[51638]: cluster 2023-12-16T07:48:11.261456+0000 osd.1 (osd.1) 6 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:12 smithi169 bash[51638]: cluster 2023-12-16T07:48:11.261998+0000 osd.2 (osd.2) 6 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:12 smithi169 bash[51638]: cluster 2023-12-16T07:48:11.264754+0000 mon.a (mon.0) 260 : cluster 0 osdmap e108: 8 total, 8 up, 8 in 2023-12-16T07:48:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:12 smithi169 bash[51638]: cluster 2023-12-16T07:48:11.264918+0000 osd.1 (osd.1) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:12 smithi169 bash[51638]: cluster 2023-12-16T07:48:11.266156+0000 osd.3 (osd.3) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:12 smithi169 bash[51638]: cluster 2023-12-16T07:48:11.266389+0000 osd.2 (osd.2) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:12 smithi169 bash[51638]: cluster 2023-12-16T07:48:11.267045+0000 osd.6 (osd.6) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:12 smithi169 bash[51638]: cluster 2023-12-16T07:48:11.268376+0000 osd.7 (osd.7) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:12 smithi169 bash[51638]: cluster 2023-12-16T07:48:11.268477+0000 osd.5 (osd.5) 7 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:12 smithi169 bash[51638]: cluster 2023-12-16T07:48:11.268519+0000 osd.4 (osd.4) 6 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:12.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:12 smithi169 bash[51638]: cluster 2023-12-16T07:48:11.268774+0000 osd.2 (osd.2) 8 : cluster 3 failed to encode map e108 with expected crc 2023-12-16T07:48:14.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:14 smithi007 bash[64479]: cluster 2023-12-16T07:48:13.160860+0000 mgr.y (mgr.25477) 163 : cluster 0 pgmap v78: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 69/630 objects degraded (10.952%) 2023-12-16T07:48:14.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:14 smithi007 bash[65864]: cluster 2023-12-16T07:48:13.160860+0000 mgr.y (mgr.25477) 163 : cluster 0 pgmap v78: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 69/630 objects degraded (10.952%) 2023-12-16T07:48:14.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:14 smithi169 bash[51638]: cluster 2023-12-16T07:48:13.160860+0000 mgr.y (mgr.25477) 163 : cluster 0 pgmap v78: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 112 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 69/630 objects degraded (10.952%) 2023-12-16T07:48:15.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:15 smithi007 bash[64479]: audit 2023-12-16T07:48:14.207261+0000 mgr.y (mgr.25477) 164 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:15.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:15 smithi007 bash[65864]: audit 2023-12-16T07:48:14.207261+0000 mgr.y (mgr.25477) 164 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:15.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:15 smithi169 bash[51638]: audit 2023-12-16T07:48:14.207261+0000 mgr.y (mgr.25477) 164 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:16.449 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:16 smithi007 bash[65864]: cluster 2023-12-16T07:48:15.162524+0000 mgr.y (mgr.25477) 165 : cluster 0 pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:48:16.449 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:16 smithi007 bash[65864]: cluster 2023-12-16T07:48:15.267354+0000 mon.a (mon.0) 261 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded) 2023-12-16T07:48:16.449 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:16 smithi007 bash[65864]: cluster 2023-12-16T07:48:15.267444+0000 mon.a (mon.0) 262 : cluster 1 Cluster is now healthy 2023-12-16T07:48:16.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:16 smithi007 bash[64479]: cluster 2023-12-16T07:48:15.162524+0000 mgr.y (mgr.25477) 165 : cluster 0 pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:48:16.449 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:16 smithi007 bash[64479]: cluster 2023-12-16T07:48:15.267354+0000 mon.a (mon.0) 261 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded) 2023-12-16T07:48:16.450 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:16 smithi007 bash[64479]: cluster 2023-12-16T07:48:15.267444+0000 mon.a (mon.0) 262 : cluster 1 Cluster is now healthy 2023-12-16T07:48:16.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:16 smithi169 bash[51638]: cluster 2023-12-16T07:48:15.162524+0000 mgr.y (mgr.25477) 165 : cluster 0 pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:48:16.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:16 smithi169 bash[51638]: cluster 2023-12-16T07:48:15.267354+0000 mon.a (mon.0) 261 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded) 2023-12-16T07:48:16.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:16 smithi169 bash[51638]: cluster 2023-12-16T07:48:15.267444+0000 mon.a (mon.0) 262 : cluster 1 Cluster is now healthy 2023-12-16T07:48:18.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:18 smithi007 bash[64479]: cluster 2023-12-16T07:48:17.163310+0000 mgr.y (mgr.25477) 166 : cluster 0 pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2023-12-16T07:48:18.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:18 smithi007 bash[65864]: cluster 2023-12-16T07:48:17.163310+0000 mgr.y (mgr.25477) 166 : cluster 0 pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2023-12-16T07:48:18.662 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:18 smithi169 bash[51638]: cluster 2023-12-16T07:48:17.163310+0000 mgr.y (mgr.25477) 166 : cluster 0 pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2023-12-16T07:48:18.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:48:18 smithi169 bash[49087]: ts=2023-12-16T07:48:18.661Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2023-12-16T07:48:18.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:48:18 smithi169 bash[49087]: ts=2023-12-16T07:48:18.665Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:48:20.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:19 smithi169 bash[51638]: cluster 2023-12-16T07:48:19.165007+0000 mgr.y (mgr.25477) 167 : cluster 0 pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-12-16T07:48:20.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:19 smithi007 bash[65864]: cluster 2023-12-16T07:48:19.165007+0000 mgr.y (mgr.25477) 167 : cluster 0 pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-12-16T07:48:20.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:19 smithi007 bash[64479]: cluster 2023-12-16T07:48:19.165007+0000 mgr.y (mgr.25477) 167 : cluster 0 pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-12-16T07:48:20.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:48:20 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:48:20] "GET /metrics HTTP/1.1" 200 38316 "" "Prometheus/2.33.4" 2023-12-16T07:48:21.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[65864]: audit 2023-12-16T07:48:20.203702+0000 mon.a (mon.0) 263 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:21.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[65864]: audit 2023-12-16T07:48:20.215006+0000 mon.a (mon.0) 264 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:21.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[65864]: audit 2023-12-16T07:48:20.217497+0000 mon.a (mon.0) 265 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:48:21.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[65864]: audit 2023-12-16T07:48:20.219485+0000 mon.a (mon.0) 266 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:48:21.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[65864]: audit 2023-12-16T07:48:20.228644+0000 mon.a (mon.0) 267 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:21.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[65864]: audit 2023-12-16T07:48:20.319400+0000 mon.a (mon.0) 268 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:48:21.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[65864]: audit 2023-12-16T07:48:20.322908+0000 mon.a (mon.0) 269 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:21.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[65864]: audit 2023-12-16T07:48:20.325818+0000 mon.a (mon.0) 270 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:21.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[65864]: audit 2023-12-16T07:48:20.328252+0000 mon.a (mon.0) 271 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:21.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[65864]: audit 2023-12-16T07:48:20.330900+0000 mon.a (mon.0) 272 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-12-16T07:48:21.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[65864]: audit 2023-12-16T07:48:20.331309+0000 mgr.y (mgr.25477) 168 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-12-16T07:48:21.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[65864]: cephadm 2023-12-16T07:48:20.333273+0000 mgr.y (mgr.25477) 169 : cephadm 1 Upgrade: osd.1 is safe to restart 2023-12-16T07:48:21.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[64479]: audit 2023-12-16T07:48:20.203702+0000 mon.a (mon.0) 263 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:21.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[64479]: audit 2023-12-16T07:48:20.215006+0000 mon.a (mon.0) 264 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:21.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[64479]: audit 2023-12-16T07:48:20.217497+0000 mon.a (mon.0) 265 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:48:21.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[64479]: audit 2023-12-16T07:48:20.219485+0000 mon.a (mon.0) 266 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:48:21.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[64479]: audit 2023-12-16T07:48:20.228644+0000 mon.a (mon.0) 267 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:21.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[64479]: audit 2023-12-16T07:48:20.319400+0000 mon.a (mon.0) 268 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:48:21.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[64479]: audit 2023-12-16T07:48:20.322908+0000 mon.a (mon.0) 269 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:21.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[64479]: audit 2023-12-16T07:48:20.325818+0000 mon.a (mon.0) 270 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:21.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[64479]: audit 2023-12-16T07:48:20.328252+0000 mon.a (mon.0) 271 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:21.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[64479]: audit 2023-12-16T07:48:20.330900+0000 mon.a (mon.0) 272 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-12-16T07:48:21.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[64479]: audit 2023-12-16T07:48:20.331309+0000 mgr.y (mgr.25477) 168 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-12-16T07:48:21.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:21 smithi007 bash[64479]: cephadm 2023-12-16T07:48:20.333273+0000 mgr.y (mgr.25477) 169 : cephadm 1 Upgrade: osd.1 is safe to restart 2023-12-16T07:48:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:21 smithi169 bash[51638]: audit 2023-12-16T07:48:20.203702+0000 mon.a (mon.0) 263 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:21 smithi169 bash[51638]: audit 2023-12-16T07:48:20.215006+0000 mon.a (mon.0) 264 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:21 smithi169 bash[51638]: audit 2023-12-16T07:48:20.217497+0000 mon.a (mon.0) 265 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:48:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:21 smithi169 bash[51638]: audit 2023-12-16T07:48:20.219485+0000 mon.a (mon.0) 266 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:48:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:21 smithi169 bash[51638]: audit 2023-12-16T07:48:20.228644+0000 mon.a (mon.0) 267 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:21.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:21 smithi169 bash[51638]: audit 2023-12-16T07:48:20.319400+0000 mon.a (mon.0) 268 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:48:21.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:21 smithi169 bash[51638]: audit 2023-12-16T07:48:20.322908+0000 mon.a (mon.0) 269 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:21.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:21 smithi169 bash[51638]: audit 2023-12-16T07:48:20.325818+0000 mon.a (mon.0) 270 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:21.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:21 smithi169 bash[51638]: audit 2023-12-16T07:48:20.328252+0000 mon.a (mon.0) 271 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:21.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:21 smithi169 bash[51638]: audit 2023-12-16T07:48:20.330900+0000 mon.a (mon.0) 272 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-12-16T07:48:21.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:21 smithi169 bash[51638]: audit 2023-12-16T07:48:20.331309+0000 mgr.y (mgr.25477) 168 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2023-12-16T07:48:21.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:21 smithi169 bash[51638]: cephadm 2023-12-16T07:48:20.333273+0000 mgr.y (mgr.25477) 169 : cephadm 1 Upgrade: osd.1 is safe to restart 2023-12-16T07:48:22.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:22 smithi007 bash[65864]: cluster 2023-12-16T07:48:21.166023+0000 mgr.y (mgr.25477) 170 : cluster 0 pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 8 op/s 2023-12-16T07:48:22.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:22 smithi007 bash[64479]: cluster 2023-12-16T07:48:21.166023+0000 mgr.y (mgr.25477) 170 : cluster 0 pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 8 op/s 2023-12-16T07:48:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:22 smithi169 bash[51638]: cluster 2023-12-16T07:48:21.166023+0000 mgr.y (mgr.25477) 170 : cluster 0 pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 8 op/s 2023-12-16T07:48:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:24 smithi169 bash[51638]: cephadm 2023-12-16T07:48:23.104076+0000 mgr.y (mgr.25477) 171 : cephadm 1 Upgrade: Updating osd.1 2023-12-16T07:48:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:24 smithi169 bash[51638]: audit 2023-12-16T07:48:23.111587+0000 mon.a (mon.0) 273 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:24 smithi169 bash[51638]: audit 2023-12-16T07:48:23.116171+0000 mon.a (mon.0) 274 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-16T07:48:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:24 smithi169 bash[51638]: audit 2023-12-16T07:48:23.117754+0000 mon.a (mon.0) 275 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:48:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:24 smithi169 bash[51638]: cephadm 2023-12-16T07:48:23.121644+0000 mgr.y (mgr.25477) 172 : cephadm 1 Deploying daemon osd.1 on smithi007 2023-12-16T07:48:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:24 smithi169 bash[51638]: cluster 2023-12-16T07:48:23.166728+0000 mgr.y (mgr.25477) 173 : cluster 0 pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-12-16T07:48:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:24 smithi169 bash[51638]: audit 2023-12-16T07:48:23.208252+0000 mon.a (mon.0) 276 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:24 smithi169 bash[51638]: audit 2023-12-16T07:48:23.210103+0000 mon.a (mon.0) 277 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:48:24.434 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:24 smithi169 bash[51638]: audit 2023-12-16T07:48:23.287102+0000 mon.a (mon.0) 278 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:24.472 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[65864]: cephadm 2023-12-16T07:48:23.104076+0000 mgr.y (mgr.25477) 171 : cephadm 1 Upgrade: Updating osd.1 2023-12-16T07:48:24.472 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[65864]: audit 2023-12-16T07:48:23.111587+0000 mon.a (mon.0) 273 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:24.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[65864]: audit 2023-12-16T07:48:23.116171+0000 mon.a (mon.0) 274 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-16T07:48:24.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[65864]: audit 2023-12-16T07:48:23.117754+0000 mon.a (mon.0) 275 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:48:24.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[65864]: cephadm 2023-12-16T07:48:23.121644+0000 mgr.y (mgr.25477) 172 : cephadm 1 Deploying daemon osd.1 on smithi007 2023-12-16T07:48:24.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[65864]: cluster 2023-12-16T07:48:23.166728+0000 mgr.y (mgr.25477) 173 : cluster 0 pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-12-16T07:48:24.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[65864]: audit 2023-12-16T07:48:23.208252+0000 mon.a (mon.0) 276 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:24.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[65864]: audit 2023-12-16T07:48:23.210103+0000 mon.a (mon.0) 277 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:48:24.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[65864]: audit 2023-12-16T07:48:23.287102+0000 mon.a (mon.0) 278 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:24.473 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[64479]: cephadm 2023-12-16T07:48:23.104076+0000 mgr.y (mgr.25477) 171 : cephadm 1 Upgrade: Updating osd.1 2023-12-16T07:48:24.473 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[64479]: audit 2023-12-16T07:48:23.111587+0000 mon.a (mon.0) 273 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:24.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[64479]: audit 2023-12-16T07:48:23.116171+0000 mon.a (mon.0) 274 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-12-16T07:48:24.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[64479]: audit 2023-12-16T07:48:23.117754+0000 mon.a (mon.0) 275 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:48:24.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[64479]: cephadm 2023-12-16T07:48:23.121644+0000 mgr.y (mgr.25477) 172 : cephadm 1 Deploying daemon osd.1 on smithi007 2023-12-16T07:48:24.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[64479]: cluster 2023-12-16T07:48:23.166728+0000 mgr.y (mgr.25477) 173 : cluster 0 pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 4.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-12-16T07:48:24.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[64479]: audit 2023-12-16T07:48:23.208252+0000 mon.a (mon.0) 276 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:24.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[64479]: audit 2023-12-16T07:48:23.210103+0000 mon.a (mon.0) 277 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:48:24.474 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:24 smithi007 bash[64479]: audit 2023-12-16T07:48:23.287102+0000 mon.a (mon.0) 278 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:25.371 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:25 smithi007 bash[64479]: audit 2023-12-16T07:48:24.215153+0000 mgr.y (mgr.25477) 174 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:25.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:25 smithi007 bash[65864]: audit 2023-12-16T07:48:24.215153+0000 mgr.y (mgr.25477) 174 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:25 smithi169 bash[51638]: audit 2023-12-16T07:48:24.215153+0000 mgr.y (mgr.25477) 174 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:26.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:26 smithi007 bash[65864]: cluster 2023-12-16T07:48:25.168516+0000 mgr.y (mgr.25477) 175 : cluster 0 pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2023-12-16T07:48:26.389 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:26 smithi007 bash[64479]: cluster 2023-12-16T07:48:25.168516+0000 mgr.y (mgr.25477) 175 : cluster 0 pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2023-12-16T07:48:26.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:26 smithi169 bash[51638]: cluster 2023-12-16T07:48:25.168516+0000 mgr.y (mgr.25477) 175 : cluster 0 pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2023-12-16T07:48:28.067 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:27 smithi007 systemd[1]: Stopping Ceph osd.1 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:48:28.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:28 smithi007 bash[64479]: cluster 2023-12-16T07:48:27.169311+0000 mgr.y (mgr.25477) 176 : cluster 0 pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2023-12-16T07:48:28.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:28 smithi007 bash[64479]: cluster 2023-12-16T07:48:28.138527+0000 mon.a (mon.0) 279 : cluster 1 osd.1 marked itself down and dead 2023-12-16T07:48:28.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:28 smithi007 bash[65864]: cluster 2023-12-16T07:48:27.169311+0000 mgr.y (mgr.25477) 176 : cluster 0 pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2023-12-16T07:48:28.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:28 smithi007 bash[65864]: cluster 2023-12-16T07:48:28.138527+0000 mon.a (mon.0) 279 : cluster 1 osd.1 marked itself down and dead 2023-12-16T07:48:28.568 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:28 smithi007 bash[36352]: debug 2023-12-16T07:48:28.133+0000 7f52da149700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:48:28.568 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:28 smithi007 bash[36352]: debug 2023-12-16T07:48:28.133+0000 7f52da149700 -1 osd.1 108 *** Got signal Terminated *** 2023-12-16T07:48:28.568 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:28 smithi007 bash[36352]: debug 2023-12-16T07:48:28.133+0000 7f52da149700 -1 osd.1 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:48:28.662 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:28 smithi169 bash[51638]: cluster 2023-12-16T07:48:27.169311+0000 mgr.y (mgr.25477) 176 : cluster 0 pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2023-12-16T07:48:28.662 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:28 smithi169 bash[51638]: cluster 2023-12-16T07:48:28.138527+0000 mon.a (mon.0) 279 : cluster 1 osd.1 marked itself down and dead 2023-12-16T07:48:28.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:48:28 smithi169 bash[49087]: ts=2023-12-16T07:48:28.661Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2023-12-16T07:48:28.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:48:28 smithi169 bash[49087]: ts=2023-12-16T07:48:28.664Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:48:29.520 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[70352]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-1 2023-12-16T07:48:29.521 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[64479]: cluster 2023-12-16T07:48:28.225368+0000 mon.a (mon.0) 280 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:48:29.521 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[64479]: cluster 2023-12-16T07:48:28.236118+0000 osd.3 (osd.3) 8 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.521 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[64479]: cluster 2023-12-16T07:48:28.239947+0000 osd.7 (osd.7) 8 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.521 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[64479]: cluster 2023-12-16T07:48:28.240797+0000 mon.a (mon.0) 281 : cluster 0 osdmap e109: 8 total, 7 up, 8 in 2023-12-16T07:48:29.521 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[64479]: cluster 2023-12-16T07:48:28.241600+0000 osd.2 (osd.2) 9 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.521 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[64479]: cluster 2023-12-16T07:48:28.241782+0000 osd.3 (osd.3) 9 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.521 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[64479]: cluster 2023-12-16T07:48:28.243597+0000 osd.4 (osd.4) 7 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.521 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[64479]: cluster 2023-12-16T07:48:28.243933+0000 osd.6 (osd.6) 8 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.521 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[64479]: cluster 2023-12-16T07:48:28.244435+0000 osd.5 (osd.5) 8 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.521 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[65864]: cluster 2023-12-16T07:48:28.225368+0000 mon.a (mon.0) 280 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:48:29.522 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[65864]: cluster 2023-12-16T07:48:28.236118+0000 osd.3 (osd.3) 8 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.522 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[65864]: cluster 2023-12-16T07:48:28.239947+0000 osd.7 (osd.7) 8 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.522 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[65864]: cluster 2023-12-16T07:48:28.240797+0000 mon.a (mon.0) 281 : cluster 0 osdmap e109: 8 total, 7 up, 8 in 2023-12-16T07:48:29.522 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[65864]: cluster 2023-12-16T07:48:28.241600+0000 osd.2 (osd.2) 9 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.522 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[65864]: cluster 2023-12-16T07:48:28.241782+0000 osd.3 (osd.3) 9 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.522 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[65864]: cluster 2023-12-16T07:48:28.243597+0000 osd.4 (osd.4) 7 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.522 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[65864]: cluster 2023-12-16T07:48:28.243933+0000 osd.6 (osd.6) 8 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.522 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[65864]: cluster 2023-12-16T07:48:28.244435+0000 osd.5 (osd.5) 8 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:29 smithi169 bash[51638]: cluster 2023-12-16T07:48:28.225368+0000 mon.a (mon.0) 280 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:48:29.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:29 smithi169 bash[51638]: cluster 2023-12-16T07:48:28.236118+0000 osd.3 (osd.3) 8 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:29 smithi169 bash[51638]: cluster 2023-12-16T07:48:28.239947+0000 osd.7 (osd.7) 8 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:29 smithi169 bash[51638]: cluster 2023-12-16T07:48:28.240797+0000 mon.a (mon.0) 281 : cluster 0 osdmap e109: 8 total, 7 up, 8 in 2023-12-16T07:48:29.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:29 smithi169 bash[51638]: cluster 2023-12-16T07:48:28.241600+0000 osd.2 (osd.2) 9 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:29 smithi169 bash[51638]: cluster 2023-12-16T07:48:28.241782+0000 osd.3 (osd.3) 9 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:29 smithi169 bash[51638]: cluster 2023-12-16T07:48:28.243597+0000 osd.4 (osd.4) 7 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:29 smithi169 bash[51638]: cluster 2023-12-16T07:48:28.243933+0000 osd.6 (osd.6) 8 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:29 smithi169 bash[51638]: cluster 2023-12-16T07:48:28.244435+0000 osd.5 (osd.5) 8 : cluster 3 failed to encode map e109 with expected crc 2023-12-16T07:48:29.927 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:29 smithi007 bash[70424]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-1 2023-12-16T07:48:30.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:48:30] "GET /metrics HTTP/1.1" 200 38309 "" "Prometheus/2.33.4" 2023-12-16T07:48:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[65864]: cluster 2023-12-16T07:48:29.171215+0000 mgr.y (mgr.25477) 177 : cluster 0 pgmap v87: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 134 op/s 2023-12-16T07:48:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[65864]: cluster 2023-12-16T07:48:29.240365+0000 osd.3 (osd.3) 10 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[65864]: cluster 2023-12-16T07:48:29.240889+0000 osd.2 (osd.2) 10 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[65864]: cluster 2023-12-16T07:48:29.243271+0000 mon.a (mon.0) 282 : cluster 0 osdmap e110: 8 total, 7 up, 8 in 2023-12-16T07:48:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[65864]: cluster 2023-12-16T07:48:29.243541+0000 osd.3 (osd.3) 11 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[65864]: cluster 2023-12-16T07:48:29.245394+0000 osd.2 (osd.2) 11 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[65864]: cluster 2023-12-16T07:48:29.245547+0000 osd.6 (osd.6) 9 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[65864]: cluster 2023-12-16T07:48:29.247441+0000 osd.5 (osd.5) 9 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[65864]: cluster 2023-12-16T07:48:29.247583+0000 osd.7 (osd.7) 9 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[65864]: cluster 2023-12-16T07:48:29.247808+0000 osd.4 (osd.4) 8 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[65864]: cluster 2023-12-16T07:48:29.824143+0000 mon.a (mon.0) 283 : cluster 3 Health check failed: Reduced data availability: 10 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2023-12-16T07:48:30.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[64479]: cluster 2023-12-16T07:48:29.171215+0000 mgr.y (mgr.25477) 177 : cluster 0 pgmap v87: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 134 op/s 2023-12-16T07:48:30.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[64479]: cluster 2023-12-16T07:48:29.240365+0000 osd.3 (osd.3) 10 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[64479]: cluster 2023-12-16T07:48:29.240889+0000 osd.2 (osd.2) 10 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[64479]: cluster 2023-12-16T07:48:29.243271+0000 mon.a (mon.0) 282 : cluster 0 osdmap e110: 8 total, 7 up, 8 in 2023-12-16T07:48:30.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[64479]: cluster 2023-12-16T07:48:29.243541+0000 osd.3 (osd.3) 11 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[64479]: cluster 2023-12-16T07:48:29.245394+0000 osd.2 (osd.2) 11 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[64479]: cluster 2023-12-16T07:48:29.245547+0000 osd.6 (osd.6) 9 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[64479]: cluster 2023-12-16T07:48:29.247441+0000 osd.5 (osd.5) 9 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[64479]: cluster 2023-12-16T07:48:29.247583+0000 osd.7 (osd.7) 9 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[64479]: cluster 2023-12-16T07:48:29.247808+0000 osd.4 (osd.4) 8 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:30 smithi007 bash[64479]: cluster 2023-12-16T07:48:29.824143+0000 mon.a (mon.0) 283 : cluster 3 Health check failed: Reduced data availability: 10 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2023-12-16T07:48:30.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:30 smithi169 bash[51638]: cluster 2023-12-16T07:48:29.171215+0000 mgr.y (mgr.25477) 177 : cluster 0 pgmap v87: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 134 op/s 2023-12-16T07:48:30.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:30 smithi169 bash[51638]: cluster 2023-12-16T07:48:29.240365+0000 osd.3 (osd.3) 10 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:30 smithi169 bash[51638]: cluster 2023-12-16T07:48:29.240889+0000 osd.2 (osd.2) 10 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:30 smithi169 bash[51638]: cluster 2023-12-16T07:48:29.243271+0000 mon.a (mon.0) 282 : cluster 0 osdmap e110: 8 total, 7 up, 8 in 2023-12-16T07:48:30.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:30 smithi169 bash[51638]: cluster 2023-12-16T07:48:29.243541+0000 osd.3 (osd.3) 11 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:30 smithi169 bash[51638]: cluster 2023-12-16T07:48:29.245394+0000 osd.2 (osd.2) 11 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:30 smithi169 bash[51638]: cluster 2023-12-16T07:48:29.245547+0000 osd.6 (osd.6) 9 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:30 smithi169 bash[51638]: cluster 2023-12-16T07:48:29.247441+0000 osd.5 (osd.5) 9 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:30 smithi169 bash[51638]: cluster 2023-12-16T07:48:29.247583+0000 osd.7 (osd.7) 9 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:30 smithi169 bash[51638]: cluster 2023-12-16T07:48:29.247808+0000 osd.4 (osd.4) 8 : cluster 3 failed to encode map e110 with expected crc 2023-12-16T07:48:30.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:30 smithi169 bash[51638]: cluster 2023-12-16T07:48:29.824143+0000 mon.a (mon.0) 283 : cluster 3 Health check failed: Reduced data availability: 10 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2023-12-16T07:48:31.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:31 smithi007 bash[64479]: cluster 2023-12-16T07:48:31.241255+0000 mon.a (mon.0) 284 : cluster 3 Health check failed: Degraded data redundancy: 6/630 objects degraded (0.952%), 1 pg degraded (PG_DEGRADED) 2023-12-16T07:48:31.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:31 smithi007 bash[65864]: cluster 2023-12-16T07:48:31.241255+0000 mon.a (mon.0) 284 : cluster 3 Health check failed: Degraded data redundancy: 6/630 objects degraded (0.952%), 1 pg degraded (PG_DEGRADED) 2023-12-16T07:48:31.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:31 smithi169 bash[51638]: cluster 2023-12-16T07:48:31.241255+0000 mon.a (mon.0) 284 : cluster 3 Health check failed: Degraded data redundancy: 6/630 objects degraded (0.952%), 1 pg degraded (PG_DEGRADED) 2023-12-16T07:48:32.406 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:32 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.1.service: Succeeded. 2023-12-16T07:48:32.406 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:32 smithi007 systemd[1]: Stopped Ceph osd.1 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:48:32.406 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:32 smithi007 systemd[1]: Started Ceph osd.1 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:48:32.407 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:32 smithi007 bash[65864]: cluster 2023-12-16T07:48:31.172504+0000 mgr.y (mgr.25477) 178 : cluster 0 pgmap v89: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 165 op/s; 6/630 objects degraded (0.952%) 2023-12-16T07:48:32.407 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:32 smithi007 bash[64479]: cluster 2023-12-16T07:48:31.172504+0000 mgr.y (mgr.25477) 178 : cluster 0 pgmap v89: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 165 op/s; 6/630 objects degraded (0.952%) 2023-12-16T07:48:32.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:32 smithi169 bash[51638]: cluster 2023-12-16T07:48:31.172504+0000 mgr.y (mgr.25477) 178 : cluster 0 pgmap v89: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 0 B/s wr, 165 op/s; 6/630 objects degraded (0.952%) 2023-12-16T07:48:33.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:33 smithi169 bash[51638]: audit 2023-12-16T07:48:32.384468+0000 mon.a (mon.0) 285 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:33.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:33 smithi169 bash[51638]: audit 2023-12-16T07:48:32.392584+0000 mon.a (mon.0) 286 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:33.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:33 smithi007 bash[64479]: audit 2023-12-16T07:48:32.384468+0000 mon.a (mon.0) 285 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:33.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:33 smithi007 bash[64479]: audit 2023-12-16T07:48:32.392584+0000 mon.a (mon.0) 286 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:33.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:33 smithi007 bash[65864]: audit 2023-12-16T07:48:32.384468+0000 mon.a (mon.0) 285 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:33.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:33 smithi007 bash[65864]: audit 2023-12-16T07:48:32.392584+0000 mon.a (mon.0) 286 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:34.329 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:34 smithi007 bash[70610]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-16T07:48:34.330 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:34 smithi007 bash[70610]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2023-12-16T07:48:34.330 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:34 smithi007 bash[70610]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2023-12-16T07:48:34.330 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:34 smithi007 bash[70610]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2023-12-16T07:48:34.330 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:34 smithi007 bash[70610]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2023-12-16T07:48:34.330 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:34 smithi007 bash[70610]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-16T07:48:34.330 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:34 smithi007 bash[70610]: --> ceph-volume raw activate successful for osd ID: 1 2023-12-16T07:48:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:34 smithi169 bash[51638]: cluster 2023-12-16T07:48:33.173682+0000 mgr.y (mgr.25477) 179 : cluster 0 pgmap v90: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 6/630 objects degraded (0.952%) 2023-12-16T07:48:34.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:34 smithi007 bash[64479]: cluster 2023-12-16T07:48:33.173682+0000 mgr.y (mgr.25477) 179 : cluster 0 pgmap v90: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 6/630 objects degraded (0.952%) 2023-12-16T07:48:34.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:34 smithi007 bash[65864]: cluster 2023-12-16T07:48:33.173682+0000 mgr.y (mgr.25477) 179 : cluster 0 pgmap v90: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 6/630 objects degraded (0.952%) 2023-12-16T07:48:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:35 smithi169 bash[51638]: audit 2023-12-16T07:48:34.223186+0000 mgr.y (mgr.25477) 180 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:35.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:35 smithi007 bash[64479]: audit 2023-12-16T07:48:34.223186+0000 mgr.y (mgr.25477) 180 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:35.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:35 smithi007 bash[65864]: audit 2023-12-16T07:48:34.223186+0000 mgr.y (mgr.25477) 180 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:36.411 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:36 smithi007 bash[70898]: debug 2023-12-16T07:48:36.213+0000 7f8374271700 1 -- 172.21.15.7:0/2766081768 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55740bc0b6c0 con 0x55740bc90800 2023-12-16T07:48:36.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:36 smithi169 bash[51638]: cluster 2023-12-16T07:48:35.175730+0000 mgr.y (mgr.25477) 181 : cluster 0 pgmap v91: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 144 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:36.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:36 smithi169 bash[51638]: cluster 2023-12-16T07:48:35.395744+0000 mon.a (mon.0) 287 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 14 pgs peering) 2023-12-16T07:48:36.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:36 smithi007 bash[64479]: cluster 2023-12-16T07:48:35.175730+0000 mgr.y (mgr.25477) 181 : cluster 0 pgmap v91: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 144 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:36.767 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:36 smithi007 bash[64479]: cluster 2023-12-16T07:48:35.395744+0000 mon.a (mon.0) 287 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 14 pgs peering) 2023-12-16T07:48:36.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:36 smithi007 bash[65864]: cluster 2023-12-16T07:48:35.175730+0000 mgr.y (mgr.25477) 181 : cluster 0 pgmap v91: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 144 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:36.768 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:36 smithi007 bash[65864]: cluster 2023-12-16T07:48:35.395744+0000 mon.a (mon.0) 287 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 10 pgs inactive, 14 pgs peering) 2023-12-16T07:48:37.067 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:36 smithi007 bash[70898]: debug 2023-12-16T07:48:36.761+0000 7f837d3f7640 -1 Falling back to public interface 2023-12-16T07:48:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:38 smithi169 bash[51638]: cluster 2023-12-16T07:48:37.176946+0000 mgr.y (mgr.25477) 182 : cluster 0 pgmap v92: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 129 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:38 smithi169 bash[51638]: audit 2023-12-16T07:48:38.209018+0000 mon.a (mon.0) 288 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:38 smithi169 bash[51638]: audit 2023-12-16T07:48:38.211486+0000 mon.a (mon.0) 289 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:48:38.683 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:48:38 smithi169 bash[49087]: ts=2023-12-16T07:48:38.662Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"smithi007:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.169:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2023-12-16T07:48:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:38 smithi007 bash[65864]: cluster 2023-12-16T07:48:37.176946+0000 mgr.y (mgr.25477) 182 : cluster 0 pgmap v92: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 129 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:38 smithi007 bash[65864]: audit 2023-12-16T07:48:38.209018+0000 mon.a (mon.0) 288 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:38 smithi007 bash[65864]: audit 2023-12-16T07:48:38.211486+0000 mon.a (mon.0) 289 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:48:38.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:38 smithi007 bash[64479]: cluster 2023-12-16T07:48:37.176946+0000 mgr.y (mgr.25477) 182 : cluster 0 pgmap v92: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 129 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:38.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:38 smithi007 bash[64479]: audit 2023-12-16T07:48:38.209018+0000 mon.a (mon.0) 288 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:38.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:38 smithi007 bash[64479]: audit 2023-12-16T07:48:38.211486+0000 mon.a (mon.0) 289 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:48:39.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:39 smithi169 bash[51638]: cluster 2023-12-16T07:48:38.824143+0000 mon.a (mon.0) 290 : cluster 3 Health check update: Degraded data redundancy: 84/630 objects degraded (13.333%), 22 pgs degraded (PG_DEGRADED) 2023-12-16T07:48:39.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:39 smithi007 bash[65864]: cluster 2023-12-16T07:48:38.824143+0000 mon.a (mon.0) 290 : cluster 3 Health check update: Degraded data redundancy: 84/630 objects degraded (13.333%), 22 pgs degraded (PG_DEGRADED) 2023-12-16T07:48:39.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:39 smithi007 bash[64479]: cluster 2023-12-16T07:48:38.824143+0000 mon.a (mon.0) 290 : cluster 3 Health check update: Degraded data redundancy: 84/630 objects degraded (13.333%), 22 pgs degraded (PG_DEGRADED) 2023-12-16T07:48:40.251 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:39 smithi007 bash[70898]: debug 2023-12-16T07:48:39.824+0000 7f837d3f7640 -1 osd.1 0 read_superblock omap replica is missing. 2023-12-16T07:48:40.251 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:39 smithi007 bash[70898]: debug 2023-12-16T07:48:39.872+0000 7f837d3f7640 -1 osd.1 108 log_to_monitors true 2023-12-16T07:48:40.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:48:40 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:48:40] "GET /metrics HTTP/1.1" 200 38322 "" "Prometheus/2.33.4" 2023-12-16T07:48:40.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:40 smithi007 bash[64479]: cluster 2023-12-16T07:48:39.178507+0000 mgr.y (mgr.25477) 183 : cluster 0 pgmap v93: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:40.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:40 smithi007 bash[64479]: audit 2023-12-16T07:48:39.883381+0000 mon.a (mon.0) 291 : audit 1 from='osd.1 [v2:172.21.15.7:6810/1429915755,v1:172.21.15.7:6811/1429915755]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-16T07:48:40.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:40 smithi007 bash[64479]: audit 2023-12-16T07:48:40.072220+0000 mon.a (mon.0) 292 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:40.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:40 smithi007 bash[64479]: audit 2023-12-16T07:48:40.083523+0000 mon.a (mon.0) 293 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:40.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:40 smithi007 bash[65864]: cluster 2023-12-16T07:48:39.178507+0000 mgr.y (mgr.25477) 183 : cluster 0 pgmap v93: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:40.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:40 smithi007 bash[65864]: audit 2023-12-16T07:48:39.883381+0000 mon.a (mon.0) 291 : audit 1 from='osd.1 [v2:172.21.15.7:6810/1429915755,v1:172.21.15.7:6811/1429915755]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-16T07:48:40.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:40 smithi007 bash[65864]: audit 2023-12-16T07:48:40.072220+0000 mon.a (mon.0) 292 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:40.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:40 smithi007 bash[65864]: audit 2023-12-16T07:48:40.083523+0000 mon.a (mon.0) 293 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:40.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:40 smithi169 bash[51638]: cluster 2023-12-16T07:48:39.178507+0000 mgr.y (mgr.25477) 183 : cluster 0 pgmap v93: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 142 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:40.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:40 smithi169 bash[51638]: audit 2023-12-16T07:48:39.883381+0000 mon.a (mon.0) 291 : audit 1 from='osd.1 [v2:172.21.15.7:6810/1429915755,v1:172.21.15.7:6811/1429915755]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-12-16T07:48:40.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:40 smithi169 bash[51638]: audit 2023-12-16T07:48:40.072220+0000 mon.a (mon.0) 292 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:40.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:40 smithi169 bash[51638]: audit 2023-12-16T07:48:40.083523+0000 mon.a (mon.0) 293 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:40.897 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:48:41.317 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:48:40 smithi007 bash[70898]: debug 2023-12-16T07:48:40.892+0000 7f836d866700 -1 osd.1 108 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:48:41.687 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:48:41.688 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (5m) 1s ago 12m 15.6M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:48:41.688 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (4m) 103s ago 11m 43.6M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:48:41.688 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (3m) 1s ago 10m 47.2M - 3.5 e1d6a67b021e 8d8009d0c83f 2023-12-16T07:48:41.688 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283,8765 running (3m) 103s ago 18m 433M - 19.0.0-191-g6d933094 a29c258b5d6f 5295ba35c2cd 2023-12-16T07:48:41.688 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (4m) 1s ago 20m 488M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:48:41.688 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (2m) 1s ago 20m 50.6M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 5a266b45f3ce 2023-12-16T07:48:41.688 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (117s) 103s ago 18m 23.6M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1fc9bf864dda 2023-12-16T07:48:41.688 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (2m) 1s ago 18m 41.7M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1dc91c31fc27 2023-12-16T07:48:41.688 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (5m) 1s ago 12m 11.7M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:48:41.688 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (4m) 103s ago 12m 11.3M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:48:41.689 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (37s) 1s ago 17m 49.8M 2782M 19.0.0-191-g6d933094 a29c258b5d6f a00724c6d077 2023-12-16T07:48:41.689 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (6s) 1s ago 17m - 2782M 19.0.0-191-g6d933094 a29c258b5d6f 5e90fce88079 2023-12-16T07:48:41.689 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (16m) 1s ago 16m 61.8M 2782M 17.2.0 e1d6a67b021e 1c4dd93d433d 2023-12-16T07:48:41.689 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (15m) 1s ago 15m 62.3M 2782M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:48:41.689 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (15m) 103s ago 15m 58.5M 3037M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:48:41.689 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (14m) 103s ago 14m 55.3M 3037M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:48:41.689 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (14m) 103s ago 14m 55.8M 3037M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:48:41.689 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (13m) 103s ago 13m 56.9M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:48:41.689 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (3m) 103s ago 12m 59.0M - 2.33.4 514e6a882f6e 2652ce1b8540 2023-12-16T07:48:41.689 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (10m) 1s ago 10m 87.7M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:48:41.690 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (10m) 103s ago 10m 87.4M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:48:41.690 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (10m) 1s ago 10m 87.6M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:48:41.690 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (10m) 103s ago 10m 87.0M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:48:42.161 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:48:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:48:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 3 2023-12-16T07:48:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:48:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:48:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:48:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:48:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:48:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2023-12-16T07:48:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:48:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:48:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:48:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:48:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:48:42.162 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:48:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2023-12-16T07:48:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 7 2023-12-16T07:48:42.163 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:48:42.163 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:48:42.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:41 smithi169 bash[51638]: audit 2023-12-16T07:48:40.831872+0000 mon.a (mon.0) 294 : audit 1 from='osd.1 [v2:172.21.15.7:6810/1429915755,v1:172.21.15.7:6811/1429915755]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-16T07:48:42.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:41 smithi169 bash[51638]: cluster 2023-12-16T07:48:40.833870+0000 osd.3 (osd.3) 12 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:41 smithi169 bash[51638]: cluster 2023-12-16T07:48:40.835352+0000 osd.4 (osd.4) 9 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:41 smithi169 bash[51638]: cluster 2023-12-16T07:48:40.839028+0000 mon.a (mon.0) 295 : cluster 0 osdmap e111: 8 total, 7 up, 8 in 2023-12-16T07:48:42.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:41 smithi169 bash[51638]: audit 2023-12-16T07:48:40.839661+0000 mon.a (mon.0) 296 : audit 1 from='osd.1 [v2:172.21.15.7:6810/1429915755,v1:172.21.15.7:6811/1429915755]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:48:42.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:41 smithi169 bash[51638]: audit 2023-12-16T07:48:40.878441+0000 mgr.y (mgr.25477) 184 : audit 0 from='client.44257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:42.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:41 smithi169 bash[51638]: cluster 2023-12-16T07:48:40.966218+0000 osd.2 (osd.2) 12 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:41 smithi169 bash[51638]: cluster 2023-12-16T07:48:40.967671+0000 osd.6 (osd.6) 10 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:41 smithi169 bash[51638]: cluster 2023-12-16T07:48:40.968118+0000 osd.5 (osd.5) 10 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:41 smithi169 bash[51638]: audit 2023-12-16T07:48:41.126973+0000 mon.a (mon.0) 297 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:42.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:41 smithi169 bash[51638]: audit 2023-12-16T07:48:41.135579+0000 mon.a (mon.0) 298 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:42.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:41 smithi169 bash[51638]: cluster 2023-12-16T07:48:41.179302+0000 mgr.y (mgr.25477) 185 : cluster 0 pgmap v95: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:42.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:41 smithi169 bash[51638]: audit 2023-12-16T07:48:41.274762+0000 mgr.y (mgr.25477) 186 : audit 0 from='client.54218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:42.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[64479]: audit 2023-12-16T07:48:40.831872+0000 mon.a (mon.0) 294 : audit 1 from='osd.1 [v2:172.21.15.7:6810/1429915755,v1:172.21.15.7:6811/1429915755]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-16T07:48:42.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[64479]: cluster 2023-12-16T07:48:40.833870+0000 osd.3 (osd.3) 12 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[64479]: cluster 2023-12-16T07:48:40.835352+0000 osd.4 (osd.4) 9 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[64479]: cluster 2023-12-16T07:48:40.839028+0000 mon.a (mon.0) 295 : cluster 0 osdmap e111: 8 total, 7 up, 8 in 2023-12-16T07:48:42.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[64479]: audit 2023-12-16T07:48:40.839661+0000 mon.a (mon.0) 296 : audit 1 from='osd.1 [v2:172.21.15.7:6810/1429915755,v1:172.21.15.7:6811/1429915755]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:48:42.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[64479]: audit 2023-12-16T07:48:40.878441+0000 mgr.y (mgr.25477) 184 : audit 0 from='client.44257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:42.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[64479]: cluster 2023-12-16T07:48:40.966218+0000 osd.2 (osd.2) 12 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[64479]: cluster 2023-12-16T07:48:40.967671+0000 osd.6 (osd.6) 10 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[64479]: cluster 2023-12-16T07:48:40.968118+0000 osd.5 (osd.5) 10 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[64479]: audit 2023-12-16T07:48:41.126973+0000 mon.a (mon.0) 297 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:42.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[64479]: audit 2023-12-16T07:48:41.135579+0000 mon.a (mon.0) 298 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:42.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[64479]: cluster 2023-12-16T07:48:41.179302+0000 mgr.y (mgr.25477) 185 : cluster 0 pgmap v95: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:42.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[64479]: audit 2023-12-16T07:48:41.274762+0000 mgr.y (mgr.25477) 186 : audit 0 from='client.54218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:42.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[65864]: audit 2023-12-16T07:48:40.831872+0000 mon.a (mon.0) 294 : audit 1 from='osd.1 [v2:172.21.15.7:6810/1429915755,v1:172.21.15.7:6811/1429915755]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-12-16T07:48:42.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[65864]: cluster 2023-12-16T07:48:40.833870+0000 osd.3 (osd.3) 12 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[65864]: cluster 2023-12-16T07:48:40.835352+0000 osd.4 (osd.4) 9 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[65864]: cluster 2023-12-16T07:48:40.839028+0000 mon.a (mon.0) 295 : cluster 0 osdmap e111: 8 total, 7 up, 8 in 2023-12-16T07:48:42.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[65864]: audit 2023-12-16T07:48:40.839661+0000 mon.a (mon.0) 296 : audit 1 from='osd.1 [v2:172.21.15.7:6810/1429915755,v1:172.21.15.7:6811/1429915755]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:48:42.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[65864]: audit 2023-12-16T07:48:40.878441+0000 mgr.y (mgr.25477) 184 : audit 0 from='client.44257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:42.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[65864]: cluster 2023-12-16T07:48:40.966218+0000 osd.2 (osd.2) 12 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[65864]: cluster 2023-12-16T07:48:40.967671+0000 osd.6 (osd.6) 10 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[65864]: cluster 2023-12-16T07:48:40.968118+0000 osd.5 (osd.5) 10 : cluster 3 failed to encode map e111 with expected crc 2023-12-16T07:48:42.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[65864]: audit 2023-12-16T07:48:41.126973+0000 mon.a (mon.0) 297 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:42.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[65864]: audit 2023-12-16T07:48:41.135579+0000 mon.a (mon.0) 298 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:42.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[65864]: cluster 2023-12-16T07:48:41.179302+0000 mgr.y (mgr.25477) 185 : cluster 0 pgmap v95: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:42.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:41 smithi007 bash[65864]: audit 2023-12-16T07:48:41.274762+0000 mgr.y (mgr.25477) 186 : audit 0 from='client.54218 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:42.551 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:48:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:48:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:48:42.551 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:48:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2023-12-16T07:48:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr", 2023-12-16T07:48:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "mon" 2023-12-16T07:48:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:48:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "7/23 daemons upgraded", 2023-12-16T07:48:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading osd daemons", 2023-12-16T07:48:42.552 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:48:42.552 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:48:43.026 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_WARN Degraded data redundancy: 84/630 objects degraded (13.333%), 22 pgs degraded 2023-12-16T07:48:43.027 INFO:teuthology.orchestra.run.smithi007.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 84/630 objects degraded (13.333%), 22 pgs degraded 2023-12-16T07:48:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.6 is active+undersized+degraded, acting [6,4] 2023-12-16T07:48:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2023-12-16T07:48:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.a is active+undersized+degraded, acting [3,2] 2023-12-16T07:48:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.11 is active+undersized+degraded, acting [6,4] 2023-12-16T07:48:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2023-12-16T07:48:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2023-12-16T07:48:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2023-12-16T07:48:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2023-12-16T07:48:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2023-12-16T07:48:43.027 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2023-12-16T07:48:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2023-12-16T07:48:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2023-12-16T07:48:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2023-12-16T07:48:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2023-12-16T07:48:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2023-12-16T07:48:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.9 is active+undersized+degraded, acting [4,3] 2023-12-16T07:48:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.d is active+undersized+degraded, acting [4,2] 2023-12-16T07:48:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.e is active+undersized+degraded, acting [4,6] 2023-12-16T07:48:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2023-12-16T07:48:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.1f is active+undersized+degraded, acting [6,5] 2023-12-16T07:48:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: pg 5.1d is active+undersized+degraded, acting [4,0] 2023-12-16T07:48:43.028 INFO:teuthology.orchestra.run.smithi007.stdout: pg 6.1a is active+undersized+degraded, acting [4,5] 2023-12-16T07:48:43.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:42 smithi169 bash[51638]: audit 2023-12-16T07:48:41.661068+0000 mgr.y (mgr.25477) 187 : audit 0 from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:43.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:42 smithi169 bash[51638]: cluster 2023-12-16T07:48:41.833322+0000 mon.a (mon.0) 299 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:48:43.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:42 smithi169 bash[51638]: cluster 2023-12-16T07:48:41.843415+0000 osd.2 (osd.2) 13 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:42 smithi169 bash[51638]: cluster 2023-12-16T07:48:41.845735+0000 mon.a (mon.0) 300 : cluster 1 osd.1 [v2:172.21.15.7:6810/1429915755,v1:172.21.15.7:6811/1429915755] boot 2023-12-16T07:48:43.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:42 smithi169 bash[51638]: cluster 2023-12-16T07:48:41.845807+0000 mon.a (mon.0) 301 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2023-12-16T07:48:43.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:42 smithi169 bash[51638]: audit 2023-12-16T07:48:41.846210+0000 mon.a (mon.0) 302 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:48:43.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:42 smithi169 bash[51638]: cluster 2023-12-16T07:48:41.846970+0000 osd.3 (osd.3) 13 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:42 smithi169 bash[51638]: cluster 2023-12-16T07:48:41.848047+0000 osd.4 (osd.4) 10 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:42 smithi169 bash[51638]: cluster 2023-12-16T07:48:41.848324+0000 osd.5 (osd.5) 11 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:42 smithi169 bash[51638]: cluster 2023-12-16T07:48:41.849294+0000 osd.6 (osd.6) 11 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:42 smithi169 bash[51638]: audit 2023-12-16T07:48:42.157332+0000 mon.a (mon.0) 303 : audit 0 from='client.? 172.21.15.7:0/4107429558' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:43.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[64479]: audit 2023-12-16T07:48:41.661068+0000 mgr.y (mgr.25477) 187 : audit 0 from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:43.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[64479]: cluster 2023-12-16T07:48:41.833322+0000 mon.a (mon.0) 299 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:48:43.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[64479]: cluster 2023-12-16T07:48:41.843415+0000 osd.2 (osd.2) 13 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[64479]: cluster 2023-12-16T07:48:41.845735+0000 mon.a (mon.0) 300 : cluster 1 osd.1 [v2:172.21.15.7:6810/1429915755,v1:172.21.15.7:6811/1429915755] boot 2023-12-16T07:48:43.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[64479]: cluster 2023-12-16T07:48:41.845807+0000 mon.a (mon.0) 301 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2023-12-16T07:48:43.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[64479]: audit 2023-12-16T07:48:41.846210+0000 mon.a (mon.0) 302 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:48:43.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[64479]: cluster 2023-12-16T07:48:41.846970+0000 osd.3 (osd.3) 13 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[64479]: cluster 2023-12-16T07:48:41.848047+0000 osd.4 (osd.4) 10 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[64479]: cluster 2023-12-16T07:48:41.848324+0000 osd.5 (osd.5) 11 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[64479]: cluster 2023-12-16T07:48:41.849294+0000 osd.6 (osd.6) 11 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[64479]: audit 2023-12-16T07:48:42.157332+0000 mon.a (mon.0) 303 : audit 0 from='client.? 172.21.15.7:0/4107429558' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:43.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[65864]: audit 2023-12-16T07:48:41.661068+0000 mgr.y (mgr.25477) 187 : audit 0 from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:43.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[65864]: cluster 2023-12-16T07:48:41.833322+0000 mon.a (mon.0) 299 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:48:43.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[65864]: cluster 2023-12-16T07:48:41.843415+0000 osd.2 (osd.2) 13 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[65864]: cluster 2023-12-16T07:48:41.845735+0000 mon.a (mon.0) 300 : cluster 1 osd.1 [v2:172.21.15.7:6810/1429915755,v1:172.21.15.7:6811/1429915755] boot 2023-12-16T07:48:43.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[65864]: cluster 2023-12-16T07:48:41.845807+0000 mon.a (mon.0) 301 : cluster 0 osdmap e112: 8 total, 8 up, 8 in 2023-12-16T07:48:43.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[65864]: audit 2023-12-16T07:48:41.846210+0000 mon.a (mon.0) 302 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-12-16T07:48:43.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[65864]: cluster 2023-12-16T07:48:41.846970+0000 osd.3 (osd.3) 13 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[65864]: cluster 2023-12-16T07:48:41.848047+0000 osd.4 (osd.4) 10 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[65864]: cluster 2023-12-16T07:48:41.848324+0000 osd.5 (osd.5) 11 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[65864]: cluster 2023-12-16T07:48:41.849294+0000 osd.6 (osd.6) 11 : cluster 3 failed to encode map e112 with expected crc 2023-12-16T07:48:43.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:42 smithi007 bash[65864]: audit 2023-12-16T07:48:42.157332+0000 mon.a (mon.0) 303 : audit 0 from='client.? 172.21.15.7:0/4107429558' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:44.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:43 smithi169 bash[51638]: audit 2023-12-16T07:48:42.548641+0000 mgr.y (mgr.25477) 188 : audit 0 from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:44.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:43 smithi169 bash[51638]: cluster 2023-12-16T07:48:42.846422+0000 osd.3 (osd.3) 14 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:43 smithi169 bash[51638]: cluster 2023-12-16T07:48:42.848491+0000 osd.5 (osd.5) 12 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:43 smithi169 bash[51638]: cluster 2023-12-16T07:48:42.849571+0000 osd.3 (osd.3) 15 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:43 smithi169 bash[51638]: cluster 2023-12-16T07:48:42.849642+0000 mon.a (mon.0) 304 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2023-12-16T07:48:44.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:43 smithi169 bash[51638]: cluster 2023-12-16T07:48:42.849971+0000 osd.2 (osd.2) 14 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:43 smithi169 bash[51638]: cluster 2023-12-16T07:48:42.851743+0000 osd.6 (osd.6) 12 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:43 smithi169 bash[51638]: cluster 2023-12-16T07:48:42.852215+0000 osd.7 (osd.7) 10 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:43 smithi169 bash[51638]: cluster 2023-12-16T07:48:42.857186+0000 osd.6 (osd.6) 13 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:43 smithi169 bash[51638]: audit 2023-12-16T07:48:43.024310+0000 mon.c (mon.1) 9 : audit 0 from='client.? 172.21.15.7:0/2886106772' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:48:44.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:43 smithi169 bash[51638]: cluster 2023-12-16T07:48:43.180294+0000 mgr.y (mgr.25477) 189 : cluster 0 pgmap v98: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:44.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[65864]: audit 2023-12-16T07:48:42.548641+0000 mgr.y (mgr.25477) 188 : audit 0 from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:44.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[65864]: cluster 2023-12-16T07:48:42.846422+0000 osd.3 (osd.3) 14 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[65864]: cluster 2023-12-16T07:48:42.848491+0000 osd.5 (osd.5) 12 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[65864]: cluster 2023-12-16T07:48:42.849571+0000 osd.3 (osd.3) 15 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[65864]: cluster 2023-12-16T07:48:42.849642+0000 mon.a (mon.0) 304 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2023-12-16T07:48:44.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[65864]: cluster 2023-12-16T07:48:42.849971+0000 osd.2 (osd.2) 14 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[65864]: cluster 2023-12-16T07:48:42.851743+0000 osd.6 (osd.6) 12 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[65864]: cluster 2023-12-16T07:48:42.852215+0000 osd.7 (osd.7) 10 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[65864]: cluster 2023-12-16T07:48:42.857186+0000 osd.6 (osd.6) 13 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[65864]: audit 2023-12-16T07:48:43.024310+0000 mon.c (mon.1) 9 : audit 0 from='client.? 172.21.15.7:0/2886106772' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:48:44.233 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[65864]: cluster 2023-12-16T07:48:43.180294+0000 mgr.y (mgr.25477) 189 : cluster 0 pgmap v98: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:44.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[64479]: audit 2023-12-16T07:48:42.548641+0000 mgr.y (mgr.25477) 188 : audit 0 from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:48:44.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[64479]: cluster 2023-12-16T07:48:42.846422+0000 osd.3 (osd.3) 14 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[64479]: cluster 2023-12-16T07:48:42.848491+0000 osd.5 (osd.5) 12 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[64479]: cluster 2023-12-16T07:48:42.849571+0000 osd.3 (osd.3) 15 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[64479]: cluster 2023-12-16T07:48:42.849642+0000 mon.a (mon.0) 304 : cluster 0 osdmap e113: 8 total, 8 up, 8 in 2023-12-16T07:48:44.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[64479]: cluster 2023-12-16T07:48:42.849971+0000 osd.2 (osd.2) 14 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[64479]: cluster 2023-12-16T07:48:42.851743+0000 osd.6 (osd.6) 12 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[64479]: cluster 2023-12-16T07:48:42.852215+0000 osd.7 (osd.7) 10 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[64479]: cluster 2023-12-16T07:48:42.857186+0000 osd.6 (osd.6) 13 : cluster 3 failed to encode map e113 with expected crc 2023-12-16T07:48:44.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[64479]: audit 2023-12-16T07:48:43.024310+0000 mon.c (mon.1) 9 : audit 0 from='client.? 172.21.15.7:0/2886106772' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:48:44.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:43 smithi007 bash[64479]: cluster 2023-12-16T07:48:43.180294+0000 mgr.y (mgr.25477) 189 : cluster 0 pgmap v98: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 42 op/s; 84/630 objects degraded (13.333%) 2023-12-16T07:48:45.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:44 smithi169 bash[51638]: audit 2023-12-16T07:48:44.229596+0000 mgr.y (mgr.25477) 190 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:45.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:44 smithi007 bash[64479]: audit 2023-12-16T07:48:44.229596+0000 mgr.y (mgr.25477) 190 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:45.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:44 smithi007 bash[65864]: audit 2023-12-16T07:48:44.229596+0000 mgr.y (mgr.25477) 190 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:46.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:45 smithi169 bash[51638]: cluster 2023-12-16T07:48:45.182060+0000 mgr.y (mgr.25477) 191 : cluster 0 pgmap v99: 161 pgs: 7 active+undersized+degraded, 14 active+undersized, 140 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s; 30/630 objects degraded (4.762%); 0 B/s, 6 objects/s recovering 2023-12-16T07:48:46.183 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:45 smithi007 bash[64479]: cluster 2023-12-16T07:48:45.182060+0000 mgr.y (mgr.25477) 191 : cluster 0 pgmap v99: 161 pgs: 7 active+undersized+degraded, 14 active+undersized, 140 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s; 30/630 objects degraded (4.762%); 0 B/s, 6 objects/s recovering 2023-12-16T07:48:46.184 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:45 smithi007 bash[65864]: cluster 2023-12-16T07:48:45.182060+0000 mgr.y (mgr.25477) 191 : cluster 0 pgmap v99: 161 pgs: 7 active+undersized+degraded, 14 active+undersized, 140 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s; 30/630 objects degraded (4.762%); 0 B/s, 6 objects/s recovering 2023-12-16T07:48:47.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:46 smithi169 bash[51638]: cluster 2023-12-16T07:48:45.856194+0000 mon.a (mon.0) 305 : cluster 3 Health check update: Degraded data redundancy: 30/630 objects degraded (4.762%), 7 pgs degraded (PG_DEGRADED) 2023-12-16T07:48:47.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:46 smithi007 bash[64479]: cluster 2023-12-16T07:48:45.856194+0000 mon.a (mon.0) 305 : cluster 3 Health check update: Degraded data redundancy: 30/630 objects degraded (4.762%), 7 pgs degraded (PG_DEGRADED) 2023-12-16T07:48:47.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:46 smithi007 bash[65864]: cluster 2023-12-16T07:48:45.856194+0000 mon.a (mon.0) 305 : cluster 3 Health check update: Degraded data redundancy: 30/630 objects degraded (4.762%), 7 pgs degraded (PG_DEGRADED) 2023-12-16T07:48:47.983 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:47 smithi007 bash[65864]: cluster 2023-12-16T07:48:47.183154+0000 mgr.y (mgr.25477) 192 : cluster 0 pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 4 op/s; 0 B/s, 8 objects/s recovering 2023-12-16T07:48:47.983 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:47 smithi007 bash[64479]: cluster 2023-12-16T07:48:47.183154+0000 mgr.y (mgr.25477) 192 : cluster 0 pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 4 op/s; 0 B/s, 8 objects/s recovering 2023-12-16T07:48:48.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:47 smithi169 bash[51638]: cluster 2023-12-16T07:48:47.183154+0000 mgr.y (mgr.25477) 192 : cluster 0 pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 4 op/s; 0 B/s, 8 objects/s recovering 2023-12-16T07:48:49.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:48 smithi169 bash[51638]: cluster 2023-12-16T07:48:47.867811+0000 mon.a (mon.0) 306 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/630 objects degraded (4.762%), 7 pgs degraded) 2023-12-16T07:48:49.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:48 smithi169 bash[51638]: cluster 2023-12-16T07:48:47.867886+0000 mon.a (mon.0) 307 : cluster 1 Cluster is now healthy 2023-12-16T07:48:49.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:48 smithi007 bash[65864]: cluster 2023-12-16T07:48:47.867811+0000 mon.a (mon.0) 306 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/630 objects degraded (4.762%), 7 pgs degraded) 2023-12-16T07:48:49.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:48 smithi007 bash[65864]: cluster 2023-12-16T07:48:47.867886+0000 mon.a (mon.0) 307 : cluster 1 Cluster is now healthy 2023-12-16T07:48:49.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:48 smithi007 bash[64479]: cluster 2023-12-16T07:48:47.867811+0000 mon.a (mon.0) 306 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 30/630 objects degraded (4.762%), 7 pgs degraded) 2023-12-16T07:48:49.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:48 smithi007 bash[64479]: cluster 2023-12-16T07:48:47.867886+0000 mon.a (mon.0) 307 : cluster 1 Cluster is now healthy 2023-12-16T07:48:50.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:49 smithi007 bash[64479]: cluster 2023-12-16T07:48:49.184758+0000 mgr.y (mgr.25477) 193 : cluster 0 pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 4.4 KiB/s rd, 4 op/s; 0 B/s, 7 objects/s recovering 2023-12-16T07:48:50.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:49 smithi007 bash[65864]: cluster 2023-12-16T07:48:49.184758+0000 mgr.y (mgr.25477) 193 : cluster 0 pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 4.4 KiB/s rd, 4 op/s; 0 B/s, 7 objects/s recovering 2023-12-16T07:48:50.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:49 smithi169 bash[51638]: cluster 2023-12-16T07:48:49.184758+0000 mgr.y (mgr.25477) 193 : cluster 0 pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 4.4 KiB/s rd, 4 op/s; 0 B/s, 7 objects/s recovering 2023-12-16T07:48:50.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:48:50 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:48:50] "GET /metrics HTTP/1.1" 200 38322 "" "Prometheus/2.33.4" 2023-12-16T07:48:52.499 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[65864]: cluster 2023-12-16T07:48:51.185578+0000 mgr.y (mgr.25477) 194 : cluster 0 pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s; 0 B/s, 6 objects/s recovering 2023-12-16T07:48:52.499 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[65864]: audit 2023-12-16T07:48:51.937673+0000 mon.a (mon.0) 308 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:52.499 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[65864]: audit 2023-12-16T07:48:51.950124+0000 mon.a (mon.0) 309 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:52.499 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[65864]: audit 2023-12-16T07:48:51.952461+0000 mon.a (mon.0) 310 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:48:52.499 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[65864]: audit 2023-12-16T07:48:51.953830+0000 mon.a (mon.0) 311 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:48:52.499 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[65864]: audit 2023-12-16T07:48:51.962168+0000 mon.a (mon.0) 312 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:52.499 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[65864]: audit 2023-12-16T07:48:52.051640+0000 mon.a (mon.0) 313 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:48:52.499 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[65864]: audit 2023-12-16T07:48:52.055155+0000 mon.a (mon.0) 314 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:52.500 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[65864]: audit 2023-12-16T07:48:52.057995+0000 mon.a (mon.0) 315 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:52.500 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[65864]: audit 2023-12-16T07:48:52.060376+0000 mon.a (mon.0) 316 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:52.500 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[65864]: audit 2023-12-16T07:48:52.062978+0000 mon.a (mon.0) 317 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-12-16T07:48:52.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[64479]: cluster 2023-12-16T07:48:51.185578+0000 mgr.y (mgr.25477) 194 : cluster 0 pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s; 0 B/s, 6 objects/s recovering 2023-12-16T07:48:52.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[64479]: audit 2023-12-16T07:48:51.937673+0000 mon.a (mon.0) 308 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:52.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[64479]: audit 2023-12-16T07:48:51.950124+0000 mon.a (mon.0) 309 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:52.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[64479]: audit 2023-12-16T07:48:51.952461+0000 mon.a (mon.0) 310 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:48:52.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[64479]: audit 2023-12-16T07:48:51.953830+0000 mon.a (mon.0) 311 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:48:52.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[64479]: audit 2023-12-16T07:48:51.962168+0000 mon.a (mon.0) 312 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:52.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[64479]: audit 2023-12-16T07:48:52.051640+0000 mon.a (mon.0) 313 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:48:52.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[64479]: audit 2023-12-16T07:48:52.055155+0000 mon.a (mon.0) 314 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:52.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[64479]: audit 2023-12-16T07:48:52.057995+0000 mon.a (mon.0) 315 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:52.501 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[64479]: audit 2023-12-16T07:48:52.060376+0000 mon.a (mon.0) 316 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:52.501 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:52 smithi007 bash[64479]: audit 2023-12-16T07:48:52.062978+0000 mon.a (mon.0) 317 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-12-16T07:48:52.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:52 smithi169 bash[51638]: cluster 2023-12-16T07:48:51.185578+0000 mgr.y (mgr.25477) 194 : cluster 0 pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s; 0 B/s, 6 objects/s recovering 2023-12-16T07:48:52.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:52 smithi169 bash[51638]: audit 2023-12-16T07:48:51.937673+0000 mon.a (mon.0) 308 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:52.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:52 smithi169 bash[51638]: audit 2023-12-16T07:48:51.950124+0000 mon.a (mon.0) 309 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:52.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:52 smithi169 bash[51638]: audit 2023-12-16T07:48:51.952461+0000 mon.a (mon.0) 310 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:48:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:52 smithi169 bash[51638]: audit 2023-12-16T07:48:51.953830+0000 mon.a (mon.0) 311 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:48:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:52 smithi169 bash[51638]: audit 2023-12-16T07:48:51.962168+0000 mon.a (mon.0) 312 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:52 smithi169 bash[51638]: audit 2023-12-16T07:48:52.051640+0000 mon.a (mon.0) 313 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:48:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:52 smithi169 bash[51638]: audit 2023-12-16T07:48:52.055155+0000 mon.a (mon.0) 314 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:52 smithi169 bash[51638]: audit 2023-12-16T07:48:52.057995+0000 mon.a (mon.0) 315 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:52 smithi169 bash[51638]: audit 2023-12-16T07:48:52.060376+0000 mon.a (mon.0) 316 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:48:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:52 smithi169 bash[51638]: audit 2023-12-16T07:48:52.062978+0000 mon.a (mon.0) 317 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-12-16T07:48:53.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:53 smithi007 bash[64479]: audit 2023-12-16T07:48:52.063302+0000 mgr.y (mgr.25477) 195 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-12-16T07:48:53.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:53 smithi007 bash[64479]: cephadm 2023-12-16T07:48:52.064973+0000 mgr.y (mgr.25477) 196 : cephadm 1 Upgrade: osd.2 is safe to restart 2023-12-16T07:48:53.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:53 smithi007 bash[64479]: audit 2023-12-16T07:48:53.209884+0000 mon.a (mon.0) 318 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:53.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:53 smithi007 bash[64479]: audit 2023-12-16T07:48:53.212374+0000 mon.a (mon.0) 319 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:48:53.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:53 smithi007 bash[65864]: audit 2023-12-16T07:48:52.063302+0000 mgr.y (mgr.25477) 195 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-12-16T07:48:53.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:53 smithi007 bash[65864]: cephadm 2023-12-16T07:48:52.064973+0000 mgr.y (mgr.25477) 196 : cephadm 1 Upgrade: osd.2 is safe to restart 2023-12-16T07:48:53.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:53 smithi007 bash[65864]: audit 2023-12-16T07:48:53.209884+0000 mon.a (mon.0) 318 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:53.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:53 smithi007 bash[65864]: audit 2023-12-16T07:48:53.212374+0000 mon.a (mon.0) 319 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:48:53.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:53 smithi169 bash[51638]: audit 2023-12-16T07:48:52.063302+0000 mgr.y (mgr.25477) 195 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2023-12-16T07:48:53.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:53 smithi169 bash[51638]: cephadm 2023-12-16T07:48:52.064973+0000 mgr.y (mgr.25477) 196 : cephadm 1 Upgrade: osd.2 is safe to restart 2023-12-16T07:48:53.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:53 smithi169 bash[51638]: audit 2023-12-16T07:48:53.209884+0000 mon.a (mon.0) 318 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:53.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:53 smithi169 bash[51638]: audit 2023-12-16T07:48:53.212374+0000 mon.a (mon.0) 319 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:48:54.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:54 smithi007 bash[64479]: cluster 2023-12-16T07:48:53.186263+0000 mgr.y (mgr.25477) 197 : cluster 0 pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s; 0 B/s, 5 objects/s recovering 2023-12-16T07:48:54.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:54 smithi007 bash[64479]: audit 2023-12-16T07:48:53.326802+0000 mon.a (mon.0) 320 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:54.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:54 smithi007 bash[65864]: cluster 2023-12-16T07:48:53.186263+0000 mgr.y (mgr.25477) 197 : cluster 0 pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s; 0 B/s, 5 objects/s recovering 2023-12-16T07:48:54.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:54 smithi007 bash[65864]: audit 2023-12-16T07:48:53.326802+0000 mon.a (mon.0) 320 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:54.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:54 smithi169 bash[51638]: cluster 2023-12-16T07:48:53.186263+0000 mgr.y (mgr.25477) 197 : cluster 0 pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s; 0 B/s, 5 objects/s recovering 2023-12-16T07:48:54.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:54 smithi169 bash[51638]: audit 2023-12-16T07:48:53.326802+0000 mon.a (mon.0) 320 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:55.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:55 smithi007 bash[64479]: audit 2023-12-16T07:48:54.239551+0000 mgr.y (mgr.25477) 198 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:55.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:55 smithi007 bash[64479]: audit 2023-12-16T07:48:54.706101+0000 mon.a (mon.0) 321 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:55.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:55 smithi007 bash[64479]: audit 2023-12-16T07:48:54.710928+0000 mon.a (mon.0) 322 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-16T07:48:55.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:55 smithi007 bash[64479]: audit 2023-12-16T07:48:54.712489+0000 mon.a (mon.0) 323 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:48:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:55 smithi007 bash[65864]: audit 2023-12-16T07:48:54.239551+0000 mgr.y (mgr.25477) 198 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:55 smithi007 bash[65864]: audit 2023-12-16T07:48:54.706101+0000 mon.a (mon.0) 321 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:55 smithi007 bash[65864]: audit 2023-12-16T07:48:54.710928+0000 mon.a (mon.0) 322 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-16T07:48:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:55 smithi007 bash[65864]: audit 2023-12-16T07:48:54.712489+0000 mon.a (mon.0) 323 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:48:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:55 smithi169 bash[51638]: audit 2023-12-16T07:48:54.239551+0000 mgr.y (mgr.25477) 198 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:48:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:55 smithi169 bash[51638]: audit 2023-12-16T07:48:54.706101+0000 mon.a (mon.0) 321 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:48:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:55 smithi169 bash[51638]: audit 2023-12-16T07:48:54.710928+0000 mon.a (mon.0) 322 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-12-16T07:48:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:55 smithi169 bash[51638]: audit 2023-12-16T07:48:54.712489+0000 mon.a (mon.0) 323 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:48:56.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:56 smithi007 bash[65864]: cephadm 2023-12-16T07:48:54.698879+0000 mgr.y (mgr.25477) 199 : cephadm 1 Upgrade: Updating osd.2 2023-12-16T07:48:56.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:56 smithi007 bash[65864]: cephadm 2023-12-16T07:48:54.718001+0000 mgr.y (mgr.25477) 200 : cephadm 1 Deploying daemon osd.2 on smithi007 2023-12-16T07:48:56.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:56 smithi007 bash[65864]: cluster 2023-12-16T07:48:55.187846+0000 mgr.y (mgr.25477) 201 : cluster 0 pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 3 op/s; 0 B/s, 4 objects/s recovering 2023-12-16T07:48:56.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:56 smithi007 bash[64479]: cephadm 2023-12-16T07:48:54.698879+0000 mgr.y (mgr.25477) 199 : cephadm 1 Upgrade: Updating osd.2 2023-12-16T07:48:56.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:56 smithi007 bash[64479]: cephadm 2023-12-16T07:48:54.718001+0000 mgr.y (mgr.25477) 200 : cephadm 1 Deploying daemon osd.2 on smithi007 2023-12-16T07:48:56.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:56 smithi007 bash[64479]: cluster 2023-12-16T07:48:55.187846+0000 mgr.y (mgr.25477) 201 : cluster 0 pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 3 op/s; 0 B/s, 4 objects/s recovering 2023-12-16T07:48:56.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:56 smithi169 bash[51638]: cephadm 2023-12-16T07:48:54.698879+0000 mgr.y (mgr.25477) 199 : cephadm 1 Upgrade: Updating osd.2 2023-12-16T07:48:56.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:56 smithi169 bash[51638]: cephadm 2023-12-16T07:48:54.718001+0000 mgr.y (mgr.25477) 200 : cephadm 1 Deploying daemon osd.2 on smithi007 2023-12-16T07:48:56.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:56 smithi169 bash[51638]: cluster 2023-12-16T07:48:55.187846+0000 mgr.y (mgr.25477) 201 : cluster 0 pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 3 op/s; 0 B/s, 4 objects/s recovering 2023-12-16T07:48:58.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:58 smithi169 bash[51638]: cluster 2023-12-16T07:48:57.188799+0000 mgr.y (mgr.25477) 202 : cluster 0 pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 2 op/s; 0 B/s, 1 objects/s recovering 2023-12-16T07:48:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:58 smithi007 bash[65864]: cluster 2023-12-16T07:48:57.188799+0000 mgr.y (mgr.25477) 202 : cluster 0 pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 2 op/s; 0 B/s, 1 objects/s recovering 2023-12-16T07:48:58.691 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:58 smithi007 bash[64479]: cluster 2023-12-16T07:48:57.188799+0000 mgr.y (mgr.25477) 202 : cluster 0 pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 2 op/s; 0 B/s, 1 objects/s recovering 2023-12-16T07:49:00.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:48:59 smithi007 bash[65864]: cluster 2023-12-16T07:48:59.190263+0000 mgr.y (mgr.25477) 203 : cluster 0 pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:49:00.174 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:48:59 smithi007 bash[64479]: cluster 2023-12-16T07:48:59.190263+0000 mgr.y (mgr.25477) 203 : cluster 0 pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:49:00.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:48:59 smithi169 bash[51638]: cluster 2023-12-16T07:48:59.190263+0000 mgr.y (mgr.25477) 203 : cluster 0 pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:49:00.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:49:00 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:49:00] "GET /metrics HTTP/1.1" 200 38330 "" "Prometheus/2.33.4" 2023-12-16T07:49:00.567 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:00 smithi007 systemd[1]: Stopping Ceph osd.2 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:49:00.568 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:00 smithi007 bash[39502]: debug 2023-12-16T07:49:00.360+0000 7f492c4ef700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:49:00.568 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:00 smithi007 bash[39502]: debug 2023-12-16T07:49:00.360+0000 7f492c4ef700 -1 osd.2 113 *** Got signal Terminated *** 2023-12-16T07:49:00.568 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:00 smithi007 bash[39502]: debug 2023-12-16T07:49:00.360+0000 7f492c4ef700 -1 osd.2 113 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:49:01.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:00 smithi169 bash[51638]: cluster 2023-12-16T07:49:00.366396+0000 mon.a (mon.0) 324 : cluster 1 osd.2 marked itself down and dead 2023-12-16T07:49:01.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:00 smithi007 bash[65864]: cluster 2023-12-16T07:49:00.366396+0000 mon.a (mon.0) 324 : cluster 1 osd.2 marked itself down and dead 2023-12-16T07:49:01.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:00 smithi007 bash[64479]: cluster 2023-12-16T07:49:00.366396+0000 mon.a (mon.0) 324 : cluster 1 osd.2 marked itself down and dead 2023-12-16T07:49:02.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[65864]: cluster 2023-12-16T07:49:00.843886+0000 mon.a (mon.0) 325 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:49:02.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[65864]: cluster 2023-12-16T07:49:00.854177+0000 osd.3 (osd.3) 16 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[65864]: cluster 2023-12-16T07:49:00.858156+0000 mon.a (mon.0) 326 : cluster 0 osdmap e114: 8 total, 7 up, 8 in 2023-12-16T07:49:02.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[65864]: cluster 2023-12-16T07:49:00.861561+0000 osd.4 (osd.4) 11 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[65864]: cluster 2023-12-16T07:49:00.865297+0000 osd.5 (osd.5) 13 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[65864]: cluster 2023-12-16T07:49:00.866640+0000 osd.7 (osd.7) 11 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[65864]: cluster 2023-12-16T07:49:00.869255+0000 osd.6 (osd.6) 14 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[65864]: cluster 2023-12-16T07:49:01.191565+0000 mgr.y (mgr.25477) 204 : cluster 0 pgmap v108: 161 pgs: 8 peering, 10 stale+active+clean, 143 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:49:02.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[64479]: cluster 2023-12-16T07:49:00.843886+0000 mon.a (mon.0) 325 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:49:02.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[64479]: cluster 2023-12-16T07:49:00.854177+0000 osd.3 (osd.3) 16 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[64479]: cluster 2023-12-16T07:49:00.858156+0000 mon.a (mon.0) 326 : cluster 0 osdmap e114: 8 total, 7 up, 8 in 2023-12-16T07:49:02.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[64479]: cluster 2023-12-16T07:49:00.861561+0000 osd.4 (osd.4) 11 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[64479]: cluster 2023-12-16T07:49:00.865297+0000 osd.5 (osd.5) 13 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[64479]: cluster 2023-12-16T07:49:00.866640+0000 osd.7 (osd.7) 11 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[64479]: cluster 2023-12-16T07:49:00.869255+0000 osd.6 (osd.6) 14 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:01 smithi007 bash[64479]: cluster 2023-12-16T07:49:01.191565+0000 mgr.y (mgr.25477) 204 : cluster 0 pgmap v108: 161 pgs: 8 peering, 10 stale+active+clean, 143 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:49:02.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:01 smithi169 bash[51638]: cluster 2023-12-16T07:49:00.843886+0000 mon.a (mon.0) 325 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:49:02.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:01 smithi169 bash[51638]: cluster 2023-12-16T07:49:00.854177+0000 osd.3 (osd.3) 16 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:01 smithi169 bash[51638]: cluster 2023-12-16T07:49:00.858156+0000 mon.a (mon.0) 326 : cluster 0 osdmap e114: 8 total, 7 up, 8 in 2023-12-16T07:49:02.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:01 smithi169 bash[51638]: cluster 2023-12-16T07:49:00.861561+0000 osd.4 (osd.4) 11 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:01 smithi169 bash[51638]: cluster 2023-12-16T07:49:00.865297+0000 osd.5 (osd.5) 13 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:01 smithi169 bash[51638]: cluster 2023-12-16T07:49:00.866640+0000 osd.7 (osd.7) 11 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:01 smithi169 bash[51638]: cluster 2023-12-16T07:49:00.869255+0000 osd.6 (osd.6) 14 : cluster 3 failed to encode map e114 with expected crc 2023-12-16T07:49:02.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:01 smithi169 bash[51638]: cluster 2023-12-16T07:49:01.191565+0000 mgr.y (mgr.25477) 204 : cluster 0 pgmap v108: 161 pgs: 8 peering, 10 stale+active+clean, 143 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:49:02.445 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[72515]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-2 2023-12-16T07:49:02.817 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[72590]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-2 2023-12-16T07:49:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:02 smithi169 bash[51638]: cluster 2023-12-16T07:49:01.864988+0000 mon.a (mon.0) 327 : cluster 0 osdmap e115: 8 total, 7 up, 8 in 2023-12-16T07:49:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:02 smithi169 bash[51638]: cluster 2023-12-16T07:49:01.865324+0000 osd.6 (osd.6) 15 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:02 smithi169 bash[51638]: cluster 2023-12-16T07:49:01.865563+0000 osd.3 (osd.3) 17 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:02 smithi169 bash[51638]: cluster 2023-12-16T07:49:01.865817+0000 osd.4 (osd.4) 12 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:02 smithi169 bash[51638]: cluster 2023-12-16T07:49:01.870371+0000 osd.7 (osd.7) 12 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:02 smithi169 bash[51638]: cluster 2023-12-16T07:49:01.870436+0000 osd.5 (osd.5) 14 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:02 smithi169 bash[51638]: cluster 2023-12-16T07:49:01.870701+0000 osd.6 (osd.6) 16 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:02 smithi169 bash[51638]: cluster 2023-12-16T07:49:01.871104+0000 osd.4 (osd.4) 13 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:02 smithi169 bash[51638]: cluster 2023-12-16T07:49:01.872725+0000 osd.6 (osd.6) 17 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.185 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:02 smithi169 bash[51638]: cluster 2023-12-16T07:49:01.874100+0000 osd.4 (osd.4) 14 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[65864]: cluster 2023-12-16T07:49:01.864988+0000 mon.a (mon.0) 327 : cluster 0 osdmap e115: 8 total, 7 up, 8 in 2023-12-16T07:49:03.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[65864]: cluster 2023-12-16T07:49:01.865324+0000 osd.6 (osd.6) 15 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[65864]: cluster 2023-12-16T07:49:01.865563+0000 osd.3 (osd.3) 17 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[65864]: cluster 2023-12-16T07:49:01.865817+0000 osd.4 (osd.4) 12 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[65864]: cluster 2023-12-16T07:49:01.870371+0000 osd.7 (osd.7) 12 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[65864]: cluster 2023-12-16T07:49:01.870436+0000 osd.5 (osd.5) 14 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[65864]: cluster 2023-12-16T07:49:01.870701+0000 osd.6 (osd.6) 16 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[65864]: cluster 2023-12-16T07:49:01.871104+0000 osd.4 (osd.4) 13 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[65864]: cluster 2023-12-16T07:49:01.872725+0000 osd.6 (osd.6) 17 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[65864]: cluster 2023-12-16T07:49:01.874100+0000 osd.4 (osd.4) 14 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[64479]: cluster 2023-12-16T07:49:01.864988+0000 mon.a (mon.0) 327 : cluster 0 osdmap e115: 8 total, 7 up, 8 in 2023-12-16T07:49:03.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[64479]: cluster 2023-12-16T07:49:01.865324+0000 osd.6 (osd.6) 15 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[64479]: cluster 2023-12-16T07:49:01.865563+0000 osd.3 (osd.3) 17 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[64479]: cluster 2023-12-16T07:49:01.865817+0000 osd.4 (osd.4) 12 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[64479]: cluster 2023-12-16T07:49:01.870371+0000 osd.7 (osd.7) 12 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[64479]: cluster 2023-12-16T07:49:01.870436+0000 osd.5 (osd.5) 14 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[64479]: cluster 2023-12-16T07:49:01.870701+0000 osd.6 (osd.6) 16 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[64479]: cluster 2023-12-16T07:49:01.871104+0000 osd.4 (osd.4) 13 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[64479]: cluster 2023-12-16T07:49:01.872725+0000 osd.6 (osd.6) 17 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:03.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:02 smithi007 bash[64479]: cluster 2023-12-16T07:49:01.874100+0000 osd.4 (osd.4) 14 : cluster 3 failed to encode map e115 with expected crc 2023-12-16T07:49:04.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:03 smithi169 bash[51638]: cluster 2023-12-16T07:49:03.192359+0000 mgr.y (mgr.25477) 205 : cluster 0 pgmap v110: 161 pgs: 8 peering, 10 stale+active+clean, 143 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-12-16T07:49:04.191 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:03 smithi007 bash[65864]: cluster 2023-12-16T07:49:03.192359+0000 mgr.y (mgr.25477) 205 : cluster 0 pgmap v110: 161 pgs: 8 peering, 10 stale+active+clean, 143 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-12-16T07:49:04.191 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:03 smithi007 bash[64479]: cluster 2023-12-16T07:49:03.192359+0000 mgr.y (mgr.25477) 205 : cluster 0 pgmap v110: 161 pgs: 8 peering, 10 stale+active+clean, 143 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2023-12-16T07:49:05.176 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:04 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.2.service: Succeeded. 2023-12-16T07:49:05.177 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:04 smithi007 systemd[1]: Stopped Ceph osd.2 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:49:05.177 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:04 smithi007 bash[65864]: audit 2023-12-16T07:49:04.247277+0000 mgr.y (mgr.25477) 206 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:05.177 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:04 smithi007 bash[64479]: audit 2023-12-16T07:49:04.247277+0000 mgr.y (mgr.25477) 206 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:05.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:04 smithi169 bash[51638]: audit 2023-12-16T07:49:04.247277+0000 mgr.y (mgr.25477) 206 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:05.567 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:05 smithi007 systemd[1]: Started Ceph osd.2 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:49:06.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[64479]: cluster 2023-12-16T07:49:05.193485+0000 mgr.y (mgr.25477) 207 : cluster 0 pgmap v111: 161 pgs: 12 active+undersized+degraded, 8 peering, 27 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 37/630 objects degraded (5.873%) 2023-12-16T07:49:06.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[64479]: audit 2023-12-16T07:49:05.222445+0000 mon.a (mon.0) 328 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:06.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[64479]: audit 2023-12-16T07:49:05.231570+0000 mon.a (mon.0) 329 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:06.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[64479]: cluster 2023-12-16T07:49:05.875168+0000 mon.a (mon.0) 330 : cluster 3 Health check failed: Degraded data redundancy: 37/630 objects degraded (5.873%), 12 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:06.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[65864]: cluster 2023-12-16T07:49:05.193485+0000 mgr.y (mgr.25477) 207 : cluster 0 pgmap v111: 161 pgs: 12 active+undersized+degraded, 8 peering, 27 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 37/630 objects degraded (5.873%) 2023-12-16T07:49:06.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[65864]: audit 2023-12-16T07:49:05.222445+0000 mon.a (mon.0) 328 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:06.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[65864]: audit 2023-12-16T07:49:05.231570+0000 mon.a (mon.0) 329 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:06.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[65864]: cluster 2023-12-16T07:49:05.875168+0000 mon.a (mon.0) 330 : cluster 3 Health check failed: Degraded data redundancy: 37/630 objects degraded (5.873%), 12 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:06 smithi169 bash[51638]: cluster 2023-12-16T07:49:05.193485+0000 mgr.y (mgr.25477) 207 : cluster 0 pgmap v111: 161 pgs: 12 active+undersized+degraded, 8 peering, 27 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 37/630 objects degraded (5.873%) 2023-12-16T07:49:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:06 smithi169 bash[51638]: audit 2023-12-16T07:49:05.222445+0000 mon.a (mon.0) 328 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:06 smithi169 bash[51638]: audit 2023-12-16T07:49:05.231570+0000 mon.a (mon.0) 329 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:06.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:06 smithi169 bash[51638]: cluster 2023-12-16T07:49:05.875168+0000 mon.a (mon.0) 330 : cluster 3 Health check failed: Degraded data redundancy: 37/630 objects degraded (5.873%), 12 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:07.124 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[72761]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-16T07:49:07.124 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[72761]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2023-12-16T07:49:07.124 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[72761]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2023-12-16T07:49:07.124 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[72761]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-12-16T07:49:07.124 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[72761]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2023-12-16T07:49:07.124 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[72761]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-16T07:49:07.125 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:06 smithi007 bash[72761]: --> ceph-volume raw activate successful for osd ID: 2 2023-12-16T07:49:08.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:08 smithi007 bash[64479]: cluster 2023-12-16T07:49:07.194687+0000 mgr.y (mgr.25477) 208 : cluster 0 pgmap v112: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 57/630 objects degraded (9.048%) 2023-12-16T07:49:08.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:08 smithi007 bash[64479]: audit 2023-12-16T07:49:08.209784+0000 mon.a (mon.0) 331 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:08.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:08 smithi007 bash[64479]: audit 2023-12-16T07:49:08.212077+0000 mon.a (mon.0) 332 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:49:08.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:08 smithi007 bash[65864]: cluster 2023-12-16T07:49:07.194687+0000 mgr.y (mgr.25477) 208 : cluster 0 pgmap v112: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 57/630 objects degraded (9.048%) 2023-12-16T07:49:08.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:08 smithi007 bash[65864]: audit 2023-12-16T07:49:08.209784+0000 mon.a (mon.0) 331 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:08.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:08 smithi007 bash[65864]: audit 2023-12-16T07:49:08.212077+0000 mon.a (mon.0) 332 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:49:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:08 smithi169 bash[51638]: cluster 2023-12-16T07:49:07.194687+0000 mgr.y (mgr.25477) 208 : cluster 0 pgmap v112: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 57/630 objects degraded (9.048%) 2023-12-16T07:49:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:08 smithi169 bash[51638]: audit 2023-12-16T07:49:08.209784+0000 mon.a (mon.0) 331 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:08.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:08 smithi169 bash[51638]: audit 2023-12-16T07:49:08.212077+0000 mon.a (mon.0) 332 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:49:09.067 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:08 smithi007 bash[73067]: debug 2023-12-16T07:49:08.712+0000 7feb2f86b700 1 -- 172.21.15.7:0/830231435 <== mon.1 v2:172.21.15.7:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56074314f6c0 con 0x5607431d5000 2023-12-16T07:49:09.567 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:09 smithi007 bash[73067]: debug 2023-12-16T07:49:09.264+0000 7feb389f1640 -1 Falling back to public interface 2023-12-16T07:49:10.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:09 smithi169 bash[51638]: cluster 2023-12-16T07:49:09.196307+0000 mgr.y (mgr.25477) 209 : cluster 0 pgmap v113: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 368 B/s rd, 0 op/s; 57/630 objects degraded (9.048%) 2023-12-16T07:49:10.250 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:09 smithi007 bash[65864]: cluster 2023-12-16T07:49:09.196307+0000 mgr.y (mgr.25477) 209 : cluster 0 pgmap v113: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 368 B/s rd, 0 op/s; 57/630 objects degraded (9.048%) 2023-12-16T07:49:10.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:09 smithi007 bash[64479]: cluster 2023-12-16T07:49:09.196307+0000 mgr.y (mgr.25477) 209 : cluster 0 pgmap v113: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 368 B/s rd, 0 op/s; 57/630 objects degraded (9.048%) 2023-12-16T07:49:10.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:49:10 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:49:10] "GET /metrics HTTP/1.1" 200 38324 "" "Prometheus/2.33.4" 2023-12-16T07:49:12.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:12 smithi007 bash[65864]: cluster 2023-12-16T07:49:11.197598+0000 mgr.y (mgr.25477) 210 : cluster 0 pgmap v114: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 57/630 objects degraded (9.048%) 2023-12-16T07:49:12.568 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:12 smithi007 bash[73067]: debug 2023-12-16T07:49:12.316+0000 7feb389f1640 -1 osd.2 0 read_superblock omap replica is missing. 2023-12-16T07:49:12.568 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:12 smithi007 bash[73067]: debug 2023-12-16T07:49:12.352+0000 7feb389f1640 -1 osd.2 113 log_to_monitors true 2023-12-16T07:49:12.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:12 smithi007 bash[64479]: cluster 2023-12-16T07:49:11.197598+0000 mgr.y (mgr.25477) 210 : cluster 0 pgmap v114: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 57/630 objects degraded (9.048%) 2023-12-16T07:49:12.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:12 smithi169 bash[51638]: cluster 2023-12-16T07:49:11.197598+0000 mgr.y (mgr.25477) 210 : cluster 0 pgmap v114: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 57/630 objects degraded (9.048%) 2023-12-16T07:49:13.476 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:49:13.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:13 smithi007 bash[65864]: audit 2023-12-16T07:49:12.363250+0000 mon.a (mon.0) 333 : audit 1 from='osd.2 [v2:172.21.15.7:6818/395168054,v1:172.21.15.7:6819/395168054]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-16T07:49:13.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:13 smithi007 bash[65864]: audit 2023-12-16T07:49:12.778791+0000 mon.a (mon.0) 334 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:13.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:13 smithi007 bash[65864]: audit 2023-12-16T07:49:12.787346+0000 mon.a (mon.0) 335 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:13.568 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:49:13 smithi007 bash[73067]: debug 2023-12-16T07:49:13.308+0000 7feb27e5e700 -1 osd.2 113 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:49:13.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:13 smithi007 bash[64479]: audit 2023-12-16T07:49:12.363250+0000 mon.a (mon.0) 333 : audit 1 from='osd.2 [v2:172.21.15.7:6818/395168054,v1:172.21.15.7:6819/395168054]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-16T07:49:13.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:13 smithi007 bash[64479]: audit 2023-12-16T07:49:12.778791+0000 mon.a (mon.0) 334 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:13.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:13 smithi007 bash[64479]: audit 2023-12-16T07:49:12.787346+0000 mon.a (mon.0) 335 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:13.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:13 smithi169 bash[51638]: audit 2023-12-16T07:49:12.363250+0000 mon.a (mon.0) 333 : audit 1 from='osd.2 [v2:172.21.15.7:6818/395168054,v1:172.21.15.7:6819/395168054]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-12-16T07:49:13.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:13 smithi169 bash[51638]: audit 2023-12-16T07:49:12.778791+0000 mon.a (mon.0) 334 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:13.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:13 smithi169 bash[51638]: audit 2023-12-16T07:49:12.787346+0000 mon.a (mon.0) 335 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:14.266 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:49:14.266 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (5m) 1s ago 12m 16.2M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:49:14.266 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (4m) 2m ago 12m 43.6M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:49:14.266 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (4m) 1s ago 10m 47.6M - 3.5 e1d6a67b021e 8d8009d0c83f 2023-12-16T07:49:14.266 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283,8765 running (3m) 2m ago 18m 433M - 19.0.0-191-g6d933094 a29c258b5d6f 5295ba35c2cd 2023-12-16T07:49:14.266 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (4m) 1s ago 20m 492M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:49:14.267 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (3m) 1s ago 21m 51.8M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 5a266b45f3ce 2023-12-16T07:49:14.267 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (2m) 2m ago 19m 23.6M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1fc9bf864dda 2023-12-16T07:49:14.267 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (2m) 1s ago 19m 43.7M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1dc91c31fc27 2023-12-16T07:49:14.267 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (5m) 1s ago 13m 11.8M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:49:14.267 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (5m) 2m ago 13m 11.3M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:49:14.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (70s) 1s ago 18m 50.8M 2782M 19.0.0-191-g6d933094 a29c258b5d6f a00724c6d077 2023-12-16T07:49:14.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (39s) 1s ago 17m 50.5M 2782M 19.0.0-191-g6d933094 a29c258b5d6f 5e90fce88079 2023-12-16T07:49:14.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (6s) 1s ago 17m - 2782M 19.0.0-191-g6d933094 a29c258b5d6f 906a30291e88 2023-12-16T07:49:14.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (16m) 1s ago 16m 63.0M 2782M 17.2.0 e1d6a67b021e 4acced671599 2023-12-16T07:49:14.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (15m) 2m ago 15m 58.5M 3037M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:49:14.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (15m) 2m ago 15m 55.3M 3037M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:49:14.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (14m) 2m ago 14m 55.8M 3037M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:49:14.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (13m) 2m ago 14m 56.9M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:49:14.268 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (4m) 2m ago 13m 59.0M - 2.33.4 514e6a882f6e 2652ce1b8540 2023-12-16T07:49:14.268 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (10m) 1s ago 10m 87.9M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:49:14.268 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (10m) 2m ago 10m 87.4M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:49:14.268 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (10m) 1s ago 10m 87.8M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:49:14.268 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (10m) 2m ago 10m 87.0M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:49:14.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[65864]: cluster 2023-12-16T07:49:13.198338+0000 mgr.y (mgr.25477) 211 : cluster 0 pgmap v115: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 271 B/s rd, 0 op/s; 57/630 objects degraded (9.048%) 2023-12-16T07:49:14.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[65864]: audit 2023-12-16T07:49:13.264785+0000 mon.a (mon.0) 336 : audit 1 from='osd.2 [v2:172.21.15.7:6818/395168054,v1:172.21.15.7:6819/395168054]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-16T07:49:14.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[65864]: cluster 2023-12-16T07:49:13.268365+0000 mon.a (mon.0) 337 : cluster 0 osdmap e116: 8 total, 7 up, 8 in 2023-12-16T07:49:14.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[65864]: cluster 2023-12-16T07:49:13.269148+0000 osd.4 (osd.4) 15 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[65864]: audit 2023-12-16T07:49:13.269198+0000 mon.a (mon.0) 338 : audit 1 from='osd.2 [v2:172.21.15.7:6818/395168054,v1:172.21.15.7:6819/395168054]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:49:14.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[65864]: cluster 2023-12-16T07:49:13.269456+0000 osd.6 (osd.6) 18 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[65864]: cluster 2023-12-16T07:49:13.333477+0000 osd.3 (osd.3) 18 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[65864]: cluster 2023-12-16T07:49:13.335987+0000 osd.5 (osd.5) 15 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[65864]: audit 2023-12-16T07:49:13.360809+0000 mon.a (mon.0) 339 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:14.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[65864]: audit 2023-12-16T07:49:13.457120+0000 mgr.y (mgr.25477) 212 : audit 0 from='client.44299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:14.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[65864]: cluster 2023-12-16T07:49:13.693087+0000 osd.7 (osd.7) 13 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[65864]: audit 2023-12-16T07:49:13.833127+0000 mon.a (mon.0) 340 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:14.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[65864]: cluster 2023-12-16T07:49:13.834571+0000 mon.a (mon.0) 341 : cluster 3 Health check update: Degraded data redundancy: 57/630 objects degraded (9.048%), 15 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:14.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[65864]: audit 2023-12-16T07:49:13.847047+0000 mon.a (mon.0) 342 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:14.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[64479]: cluster 2023-12-16T07:49:13.198338+0000 mgr.y (mgr.25477) 211 : cluster 0 pgmap v115: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 271 B/s rd, 0 op/s; 57/630 objects degraded (9.048%) 2023-12-16T07:49:14.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[64479]: audit 2023-12-16T07:49:13.264785+0000 mon.a (mon.0) 336 : audit 1 from='osd.2 [v2:172.21.15.7:6818/395168054,v1:172.21.15.7:6819/395168054]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-16T07:49:14.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[64479]: cluster 2023-12-16T07:49:13.268365+0000 mon.a (mon.0) 337 : cluster 0 osdmap e116: 8 total, 7 up, 8 in 2023-12-16T07:49:14.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[64479]: cluster 2023-12-16T07:49:13.269148+0000 osd.4 (osd.4) 15 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[64479]: audit 2023-12-16T07:49:13.269198+0000 mon.a (mon.0) 338 : audit 1 from='osd.2 [v2:172.21.15.7:6818/395168054,v1:172.21.15.7:6819/395168054]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:49:14.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[64479]: cluster 2023-12-16T07:49:13.269456+0000 osd.6 (osd.6) 18 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[64479]: cluster 2023-12-16T07:49:13.333477+0000 osd.3 (osd.3) 18 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[64479]: cluster 2023-12-16T07:49:13.335987+0000 osd.5 (osd.5) 15 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[64479]: audit 2023-12-16T07:49:13.360809+0000 mon.a (mon.0) 339 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:14.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[64479]: audit 2023-12-16T07:49:13.457120+0000 mgr.y (mgr.25477) 212 : audit 0 from='client.44299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:14.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[64479]: cluster 2023-12-16T07:49:13.693087+0000 osd.7 (osd.7) 13 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[64479]: audit 2023-12-16T07:49:13.833127+0000 mon.a (mon.0) 340 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:14.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[64479]: cluster 2023-12-16T07:49:13.834571+0000 mon.a (mon.0) 341 : cluster 3 Health check update: Degraded data redundancy: 57/630 objects degraded (9.048%), 15 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:14.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:14 smithi007 bash[64479]: audit 2023-12-16T07:49:13.847047+0000 mon.a (mon.0) 342 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:14.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:14 smithi169 bash[51638]: cluster 2023-12-16T07:49:13.198338+0000 mgr.y (mgr.25477) 211 : cluster 0 pgmap v115: 161 pgs: 15 active+undersized+degraded, 32 active+undersized, 114 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 271 B/s rd, 0 op/s; 57/630 objects degraded (9.048%) 2023-12-16T07:49:14.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:14 smithi169 bash[51638]: audit 2023-12-16T07:49:13.264785+0000 mon.a (mon.0) 336 : audit 1 from='osd.2 [v2:172.21.15.7:6818/395168054,v1:172.21.15.7:6819/395168054]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-12-16T07:49:14.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:14 smithi169 bash[51638]: cluster 2023-12-16T07:49:13.268365+0000 mon.a (mon.0) 337 : cluster 0 osdmap e116: 8 total, 7 up, 8 in 2023-12-16T07:49:14.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:14 smithi169 bash[51638]: cluster 2023-12-16T07:49:13.269148+0000 osd.4 (osd.4) 15 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:14 smithi169 bash[51638]: audit 2023-12-16T07:49:13.269198+0000 mon.a (mon.0) 338 : audit 1 from='osd.2 [v2:172.21.15.7:6818/395168054,v1:172.21.15.7:6819/395168054]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:49:14.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:14 smithi169 bash[51638]: cluster 2023-12-16T07:49:13.269456+0000 osd.6 (osd.6) 18 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:14 smithi169 bash[51638]: cluster 2023-12-16T07:49:13.333477+0000 osd.3 (osd.3) 18 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:14 smithi169 bash[51638]: cluster 2023-12-16T07:49:13.335987+0000 osd.5 (osd.5) 15 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:14 smithi169 bash[51638]: audit 2023-12-16T07:49:13.360809+0000 mon.a (mon.0) 339 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:14.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:14 smithi169 bash[51638]: audit 2023-12-16T07:49:13.457120+0000 mgr.y (mgr.25477) 212 : audit 0 from='client.44299 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:14.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:14 smithi169 bash[51638]: cluster 2023-12-16T07:49:13.693087+0000 osd.7 (osd.7) 13 : cluster 3 failed to encode map e116 with expected crc 2023-12-16T07:49:14.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:14 smithi169 bash[51638]: audit 2023-12-16T07:49:13.833127+0000 mon.a (mon.0) 340 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:14.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:14 smithi169 bash[51638]: cluster 2023-12-16T07:49:13.834571+0000 mon.a (mon.0) 341 : cluster 3 Health check update: Degraded data redundancy: 57/630 objects degraded (9.048%), 15 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:14.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:14 smithi169 bash[51638]: audit 2023-12-16T07:49:13.847047+0000 mon.a (mon.0) 342 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:14.737 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:49:14.737 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:49:14.737 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 3 2023-12-16T07:49:14.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:49:14.737 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:49:14.737 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:49:14.737 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:49:14.737 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:49:14.737 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2023-12-16T07:49:14.738 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 3 2023-12-16T07:49:14.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:49:14.738 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:49:14.738 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:49:14.738 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:49:14.738 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:49:14.738 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2023-12-16T07:49:14.738 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 8 2023-12-16T07:49:14.738 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:49:14.738 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:49:15.125 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:49:15.125 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:49:15.125 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:49:15.125 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:49:15.126 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2023-12-16T07:49:15.126 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr", 2023-12-16T07:49:15.126 INFO:teuthology.orchestra.run.smithi007.stdout: "mon" 2023-12-16T07:49:15.126 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:49:15.126 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "8/23 daemons upgraded", 2023-12-16T07:49:15.126 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading osd daemons", 2023-12-16T07:49:15.126 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:49:15.126 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:49:15.303 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:15 smithi169 bash[51638]: audit 2023-12-16T07:49:13.847530+0000 mgr.y (mgr.25477) 213 : audit 0 from='client.44305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:15.303 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:15 smithi169 bash[51638]: audit 2023-12-16T07:49:14.245580+0000 mgr.y (mgr.25477) 214 : audit 0 from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:15.303 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:15 smithi169 bash[51638]: audit 2023-12-16T07:49:14.255063+0000 mgr.y (mgr.25477) 215 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:15.303 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:15 smithi169 bash[51638]: cluster 2023-12-16T07:49:14.265425+0000 mon.a (mon.0) 343 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:49:15.303 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:15 smithi169 bash[51638]: cluster 2023-12-16T07:49:14.270471+0000 osd.6 (osd.6) 19 : cluster 3 failed to encode map e117 with expected crc 2023-12-16T07:49:15.303 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:15 smithi169 bash[51638]: cluster 2023-12-16T07:49:14.272148+0000 mon.a (mon.0) 344 : cluster 1 osd.2 [v2:172.21.15.7:6818/395168054,v1:172.21.15.7:6819/395168054] boot 2023-12-16T07:49:15.303 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:15 smithi169 bash[51638]: cluster 2023-12-16T07:49:14.272243+0000 mon.a (mon.0) 345 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2023-12-16T07:49:15.303 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:15 smithi169 bash[51638]: audit 2023-12-16T07:49:14.273249+0000 mon.a (mon.0) 346 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:49:15.303 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:15 smithi169 bash[51638]: cluster 2023-12-16T07:49:14.275329+0000 osd.4 (osd.4) 16 : cluster 3 failed to encode map e117 with expected crc 2023-12-16T07:49:15.303 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:15 smithi169 bash[51638]: cluster 2023-12-16T07:49:14.278142+0000 osd.3 (osd.3) 19 : cluster 3 failed to encode map e117 with expected crc 2023-12-16T07:49:15.303 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:15 smithi169 bash[51638]: cluster 2023-12-16T07:49:14.279127+0000 osd.4 (osd.4) 17 : cluster 3 failed to encode map e117 with expected crc 2023-12-16T07:49:15.303 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:15 smithi169 bash[51638]: audit 2023-12-16T07:49:14.734153+0000 mon.c (mon.1) 10 : audit 0 from='client.? 172.21.15.7:0/3108961953' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:15.304 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:15 smithi169 bash[51638]: cluster 2023-12-16T07:49:15.276716+0000 mon.a (mon.0) 347 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2023-12-16T07:49:15.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[65864]: audit 2023-12-16T07:49:13.847530+0000 mgr.y (mgr.25477) 213 : audit 0 from='client.44305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:15.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[65864]: audit 2023-12-16T07:49:14.245580+0000 mgr.y (mgr.25477) 214 : audit 0 from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:15.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[65864]: audit 2023-12-16T07:49:14.255063+0000 mgr.y (mgr.25477) 215 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:15.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[65864]: cluster 2023-12-16T07:49:14.265425+0000 mon.a (mon.0) 343 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:49:15.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[65864]: cluster 2023-12-16T07:49:14.270471+0000 osd.6 (osd.6) 19 : cluster 3 failed to encode map e117 with expected crc 2023-12-16T07:49:15.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[65864]: cluster 2023-12-16T07:49:14.272148+0000 mon.a (mon.0) 344 : cluster 1 osd.2 [v2:172.21.15.7:6818/395168054,v1:172.21.15.7:6819/395168054] boot 2023-12-16T07:49:15.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[65864]: cluster 2023-12-16T07:49:14.272243+0000 mon.a (mon.0) 345 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2023-12-16T07:49:15.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[65864]: audit 2023-12-16T07:49:14.273249+0000 mon.a (mon.0) 346 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:49:15.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[65864]: cluster 2023-12-16T07:49:14.275329+0000 osd.4 (osd.4) 16 : cluster 3 failed to encode map e117 with expected crc 2023-12-16T07:49:15.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[65864]: cluster 2023-12-16T07:49:14.278142+0000 osd.3 (osd.3) 19 : cluster 3 failed to encode map e117 with expected crc 2023-12-16T07:49:15.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[65864]: cluster 2023-12-16T07:49:14.279127+0000 osd.4 (osd.4) 17 : cluster 3 failed to encode map e117 with expected crc 2023-12-16T07:49:15.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[65864]: audit 2023-12-16T07:49:14.734153+0000 mon.c (mon.1) 10 : audit 0 from='client.? 172.21.15.7:0/3108961953' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:15.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[65864]: cluster 2023-12-16T07:49:15.276716+0000 mon.a (mon.0) 347 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2023-12-16T07:49:15.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[64479]: audit 2023-12-16T07:49:13.847530+0000 mgr.y (mgr.25477) 213 : audit 0 from='client.44305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:15.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[64479]: audit 2023-12-16T07:49:14.245580+0000 mgr.y (mgr.25477) 214 : audit 0 from='client.44311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:15.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[64479]: audit 2023-12-16T07:49:14.255063+0000 mgr.y (mgr.25477) 215 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:15.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[64479]: cluster 2023-12-16T07:49:14.265425+0000 mon.a (mon.0) 343 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:49:15.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[64479]: cluster 2023-12-16T07:49:14.270471+0000 osd.6 (osd.6) 19 : cluster 3 failed to encode map e117 with expected crc 2023-12-16T07:49:15.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[64479]: cluster 2023-12-16T07:49:14.272148+0000 mon.a (mon.0) 344 : cluster 1 osd.2 [v2:172.21.15.7:6818/395168054,v1:172.21.15.7:6819/395168054] boot 2023-12-16T07:49:15.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[64479]: cluster 2023-12-16T07:49:14.272243+0000 mon.a (mon.0) 345 : cluster 0 osdmap e117: 8 total, 8 up, 8 in 2023-12-16T07:49:15.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[64479]: audit 2023-12-16T07:49:14.273249+0000 mon.a (mon.0) 346 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-12-16T07:49:15.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[64479]: cluster 2023-12-16T07:49:14.275329+0000 osd.4 (osd.4) 16 : cluster 3 failed to encode map e117 with expected crc 2023-12-16T07:49:15.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[64479]: cluster 2023-12-16T07:49:14.278142+0000 osd.3 (osd.3) 19 : cluster 3 failed to encode map e117 with expected crc 2023-12-16T07:49:15.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[64479]: cluster 2023-12-16T07:49:14.279127+0000 osd.4 (osd.4) 17 : cluster 3 failed to encode map e117 with expected crc 2023-12-16T07:49:15.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[64479]: audit 2023-12-16T07:49:14.734153+0000 mon.c (mon.1) 10 : audit 0 from='client.? 172.21.15.7:0/3108961953' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:15.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:15 smithi007 bash[64479]: cluster 2023-12-16T07:49:15.276716+0000 mon.a (mon.0) 347 : cluster 0 osdmap e118: 8 total, 8 up, 8 in 2023-12-16T07:49:15.606 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_WARN Degraded data redundancy: 52/630 objects degraded (8.254%), 14 pgs degraded 2023-12-16T07:49:15.607 INFO:teuthology.orchestra.run.smithi007.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 52/630 objects degraded (8.254%), 14 pgs degraded 2023-12-16T07:49:15.607 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.1 is active+undersized+degraded, acting [3,0] 2023-12-16T07:49:15.607 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.7 is active+undersized+degraded, acting [6,7] 2023-12-16T07:49:15.607 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2023-12-16T07:49:15.607 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.c is active+undersized+degraded, acting [5,0] 2023-12-16T07:49:15.607 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.e is active+undersized+degraded, acting [3,7] 2023-12-16T07:49:15.607 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.0 is active+undersized+degraded, acting [1,6] 2023-12-16T07:49:15.607 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.4 is active+undersized+degraded, acting [1,5] 2023-12-16T07:49:15.607 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.5 is active+undersized+degraded, acting [5,3] 2023-12-16T07:49:15.607 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.9 is active+undersized+degraded, acting [4,7] 2023-12-16T07:49:15.607 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.13 is active+undersized+degraded, acting [7,4] 2023-12-16T07:49:15.607 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.1a is active+undersized+degraded, acting [4,1] 2023-12-16T07:49:15.607 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.1e is active+undersized+degraded, acting [3,6] 2023-12-16T07:49:15.608 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.d is active+undersized+degraded, acting [4,1] 2023-12-16T07:49:15.608 INFO:teuthology.orchestra.run.smithi007.stdout: pg 5.1c is active+undersized+degraded, acting [4,3] 2023-12-16T07:49:16.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[64479]: audit 2023-12-16T07:49:15.121992+0000 mgr.y (mgr.25477) 216 : audit 0 from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:16.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[64479]: cluster 2023-12-16T07:49:15.199427+0000 mgr.y (mgr.25477) 217 : cluster 0 pgmap v118: 161 pgs: 5 peering, 14 active+undersized+degraded, 28 active+undersized, 114 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 52/630 objects degraded (8.254%) 2023-12-16T07:49:16.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[64479]: cluster 2023-12-16T07:49:15.274966+0000 osd.6 (osd.6) 20 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[64479]: cluster 2023-12-16T07:49:15.275407+0000 osd.4 (osd.4) 18 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[64479]: cluster 2023-12-16T07:49:15.277028+0000 osd.3 (osd.3) 20 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[64479]: cluster 2023-12-16T07:49:15.279068+0000 osd.6 (osd.6) 21 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[64479]: cluster 2023-12-16T07:49:15.279634+0000 osd.7 (osd.7) 14 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[64479]: cluster 2023-12-16T07:49:15.280277+0000 osd.5 (osd.5) 16 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[64479]: cluster 2023-12-16T07:49:15.285689+0000 osd.5 (osd.5) 17 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[64479]: cluster 2023-12-16T07:49:15.288021+0000 osd.7 (osd.7) 15 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[64479]: cluster 2023-12-16T07:49:15.289354+0000 osd.5 (osd.5) 18 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[64479]: audit 2023-12-16T07:49:15.604526+0000 mon.a (mon.0) 348 : audit 0 from='client.? 172.21.15.7:0/1952804472' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:49:16.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[65864]: audit 2023-12-16T07:49:15.121992+0000 mgr.y (mgr.25477) 216 : audit 0 from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:16.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[65864]: cluster 2023-12-16T07:49:15.199427+0000 mgr.y (mgr.25477) 217 : cluster 0 pgmap v118: 161 pgs: 5 peering, 14 active+undersized+degraded, 28 active+undersized, 114 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 52/630 objects degraded (8.254%) 2023-12-16T07:49:16.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[65864]: cluster 2023-12-16T07:49:15.274966+0000 osd.6 (osd.6) 20 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[65864]: cluster 2023-12-16T07:49:15.275407+0000 osd.4 (osd.4) 18 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[65864]: cluster 2023-12-16T07:49:15.277028+0000 osd.3 (osd.3) 20 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[65864]: cluster 2023-12-16T07:49:15.279068+0000 osd.6 (osd.6) 21 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[65864]: cluster 2023-12-16T07:49:15.279634+0000 osd.7 (osd.7) 14 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[65864]: cluster 2023-12-16T07:49:15.280277+0000 osd.5 (osd.5) 16 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[65864]: cluster 2023-12-16T07:49:15.285689+0000 osd.5 (osd.5) 17 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[65864]: cluster 2023-12-16T07:49:15.288021+0000 osd.7 (osd.7) 15 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[65864]: cluster 2023-12-16T07:49:15.289354+0000 osd.5 (osd.5) 18 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:16 smithi007 bash[65864]: audit 2023-12-16T07:49:15.604526+0000 mon.a (mon.0) 348 : audit 0 from='client.? 172.21.15.7:0/1952804472' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:49:16.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:16 smithi169 bash[51638]: audit 2023-12-16T07:49:15.121992+0000 mgr.y (mgr.25477) 216 : audit 0 from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:16.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:16 smithi169 bash[51638]: cluster 2023-12-16T07:49:15.199427+0000 mgr.y (mgr.25477) 217 : cluster 0 pgmap v118: 161 pgs: 5 peering, 14 active+undersized+degraded, 28 active+undersized, 114 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 52/630 objects degraded (8.254%) 2023-12-16T07:49:16.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:16 smithi169 bash[51638]: cluster 2023-12-16T07:49:15.274966+0000 osd.6 (osd.6) 20 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:16 smithi169 bash[51638]: cluster 2023-12-16T07:49:15.275407+0000 osd.4 (osd.4) 18 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:16 smithi169 bash[51638]: cluster 2023-12-16T07:49:15.277028+0000 osd.3 (osd.3) 20 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:16 smithi169 bash[51638]: cluster 2023-12-16T07:49:15.279068+0000 osd.6 (osd.6) 21 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:16 smithi169 bash[51638]: cluster 2023-12-16T07:49:15.279634+0000 osd.7 (osd.7) 14 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:16 smithi169 bash[51638]: cluster 2023-12-16T07:49:15.280277+0000 osd.5 (osd.5) 16 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:16 smithi169 bash[51638]: cluster 2023-12-16T07:49:15.285689+0000 osd.5 (osd.5) 17 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:16 smithi169 bash[51638]: cluster 2023-12-16T07:49:15.288021+0000 osd.7 (osd.7) 15 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:16 smithi169 bash[51638]: cluster 2023-12-16T07:49:15.289354+0000 osd.5 (osd.5) 18 : cluster 3 failed to encode map e118 with expected crc 2023-12-16T07:49:16.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:16 smithi169 bash[51638]: audit 2023-12-16T07:49:15.604526+0000 mon.a (mon.0) 348 : audit 0 from='client.? 172.21.15.7:0/1952804472' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:49:18.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:18 smithi007 bash[64479]: cluster 2023-12-16T07:49:17.200643+0000 mgr.y (mgr.25477) 218 : cluster 0 pgmap v120: 161 pgs: 5 peering, 11 active+undersized+degraded, 25 active+undersized, 120 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 32/630 objects degraded (5.079%) 2023-12-16T07:49:18.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:18 smithi007 bash[65864]: cluster 2023-12-16T07:49:17.200643+0000 mgr.y (mgr.25477) 218 : cluster 0 pgmap v120: 161 pgs: 5 peering, 11 active+undersized+degraded, 25 active+undersized, 120 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 32/630 objects degraded (5.079%) 2023-12-16T07:49:18.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:18 smithi169 bash[51638]: cluster 2023-12-16T07:49:17.200643+0000 mgr.y (mgr.25477) 218 : cluster 0 pgmap v120: 161 pgs: 5 peering, 11 active+undersized+degraded, 25 active+undersized, 120 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 32/630 objects degraded (5.079%) 2023-12-16T07:49:19.311 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:19 smithi007 bash[65864]: cluster 2023-12-16T07:49:18.836056+0000 mon.a (mon.0) 349 : cluster 3 Health check update: Degraded data redundancy: 32/630 objects degraded (5.079%), 11 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:19.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:19 smithi007 bash[64479]: cluster 2023-12-16T07:49:18.836056+0000 mon.a (mon.0) 349 : cluster 3 Health check update: Degraded data redundancy: 32/630 objects degraded (5.079%), 11 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:19.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:19 smithi169 bash[51638]: cluster 2023-12-16T07:49:18.836056+0000 mon.a (mon.0) 349 : cluster 3 Health check update: Degraded data redundancy: 32/630 objects degraded (5.079%), 11 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:20.503 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:49:20 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:49:20] "GET /metrics HTTP/1.1" 200 38324 "" "Prometheus/2.33.4" 2023-12-16T07:49:20.503 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:20 smithi007 bash[65864]: cluster 2023-12-16T07:49:19.202186+0000 mgr.y (mgr.25477) 219 : cluster 0 pgmap v121: 161 pgs: 5 peering, 156 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-12-16T07:49:20.503 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:20 smithi007 bash[65864]: cluster 2023-12-16T07:49:19.839299+0000 mon.a (mon.0) 350 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/630 objects degraded (5.079%), 11 pgs degraded) 2023-12-16T07:49:20.503 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:20 smithi007 bash[65864]: cluster 2023-12-16T07:49:19.839373+0000 mon.a (mon.0) 351 : cluster 1 Cluster is now healthy 2023-12-16T07:49:20.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:20 smithi007 bash[64479]: cluster 2023-12-16T07:49:19.202186+0000 mgr.y (mgr.25477) 219 : cluster 0 pgmap v121: 161 pgs: 5 peering, 156 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-12-16T07:49:20.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:20 smithi007 bash[64479]: cluster 2023-12-16T07:49:19.839299+0000 mon.a (mon.0) 350 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/630 objects degraded (5.079%), 11 pgs degraded) 2023-12-16T07:49:20.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:20 smithi007 bash[64479]: cluster 2023-12-16T07:49:19.839373+0000 mon.a (mon.0) 351 : cluster 1 Cluster is now healthy 2023-12-16T07:49:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:20 smithi169 bash[51638]: cluster 2023-12-16T07:49:19.202186+0000 mgr.y (mgr.25477) 219 : cluster 0 pgmap v121: 161 pgs: 5 peering, 156 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2023-12-16T07:49:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:20 smithi169 bash[51638]: cluster 2023-12-16T07:49:19.839299+0000 mon.a (mon.0) 350 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/630 objects degraded (5.079%), 11 pgs degraded) 2023-12-16T07:49:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:20 smithi169 bash[51638]: cluster 2023-12-16T07:49:19.839373+0000 mon.a (mon.0) 351 : cluster 1 Cluster is now healthy 2023-12-16T07:49:22.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:22 smithi007 bash[65864]: cluster 2023-12-16T07:49:21.203276+0000 mgr.y (mgr.25477) 220 : cluster 0 pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:49:22.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:22 smithi007 bash[64479]: cluster 2023-12-16T07:49:21.203276+0000 mgr.y (mgr.25477) 220 : cluster 0 pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:49:22.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:22 smithi169 bash[51638]: cluster 2023-12-16T07:49:21.203276+0000 mgr.y (mgr.25477) 220 : cluster 0 pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:49:24.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:24 smithi007 bash[65864]: cluster 2023-12-16T07:49:23.203975+0000 mgr.y (mgr.25477) 221 : cluster 0 pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2023-12-16T07:49:24.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:24 smithi007 bash[65864]: audit 2023-12-16T07:49:23.211479+0000 mon.a (mon.0) 352 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:24.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:24 smithi007 bash[65864]: audit 2023-12-16T07:49:23.213794+0000 mon.a (mon.0) 353 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:49:24.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:24 smithi007 bash[65864]: audit 2023-12-16T07:49:23.379075+0000 mon.a (mon.0) 354 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:24.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:24 smithi007 bash[64479]: cluster 2023-12-16T07:49:23.203975+0000 mgr.y (mgr.25477) 221 : cluster 0 pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2023-12-16T07:49:24.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:24 smithi007 bash[64479]: audit 2023-12-16T07:49:23.211479+0000 mon.a (mon.0) 352 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:24.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:24 smithi007 bash[64479]: audit 2023-12-16T07:49:23.213794+0000 mon.a (mon.0) 353 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:49:24.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:24 smithi007 bash[64479]: audit 2023-12-16T07:49:23.379075+0000 mon.a (mon.0) 354 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:24.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:24 smithi169 bash[51638]: cluster 2023-12-16T07:49:23.203975+0000 mgr.y (mgr.25477) 221 : cluster 0 pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2023-12-16T07:49:24.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:24 smithi169 bash[51638]: audit 2023-12-16T07:49:23.211479+0000 mon.a (mon.0) 352 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:24.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:24 smithi169 bash[51638]: audit 2023-12-16T07:49:23.213794+0000 mon.a (mon.0) 353 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:49:24.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:24 smithi169 bash[51638]: audit 2023-12-16T07:49:23.379075+0000 mon.a (mon.0) 354 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:25.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[64479]: audit 2023-12-16T07:49:24.263219+0000 mgr.y (mgr.25477) 222 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[64479]: audit 2023-12-16T07:49:24.556860+0000 mon.a (mon.0) 355 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[64479]: audit 2023-12-16T07:49:24.569701+0000 mon.a (mon.0) 356 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[64479]: audit 2023-12-16T07:49:24.572307+0000 mon.a (mon.0) 357 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:49:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[64479]: audit 2023-12-16T07:49:24.573860+0000 mon.a (mon.0) 358 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:49:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[64479]: audit 2023-12-16T07:49:24.582812+0000 mon.a (mon.0) 359 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[64479]: audit 2023-12-16T07:49:24.673745+0000 mon.a (mon.0) 360 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:49:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[64479]: audit 2023-12-16T07:49:24.678576+0000 mon.a (mon.0) 361 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[64479]: audit 2023-12-16T07:49:24.683658+0000 mon.a (mon.0) 362 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[64479]: audit 2023-12-16T07:49:24.686850+0000 mon.a (mon.0) 363 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[64479]: audit 2023-12-16T07:49:24.690360+0000 mon.a (mon.0) 364 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-12-16T07:49:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[65864]: audit 2023-12-16T07:49:24.263219+0000 mgr.y (mgr.25477) 222 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[65864]: audit 2023-12-16T07:49:24.556860+0000 mon.a (mon.0) 355 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[65864]: audit 2023-12-16T07:49:24.569701+0000 mon.a (mon.0) 356 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[65864]: audit 2023-12-16T07:49:24.572307+0000 mon.a (mon.0) 357 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:49:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[65864]: audit 2023-12-16T07:49:24.573860+0000 mon.a (mon.0) 358 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:49:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[65864]: audit 2023-12-16T07:49:24.582812+0000 mon.a (mon.0) 359 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[65864]: audit 2023-12-16T07:49:24.673745+0000 mon.a (mon.0) 360 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:49:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[65864]: audit 2023-12-16T07:49:24.678576+0000 mon.a (mon.0) 361 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[65864]: audit 2023-12-16T07:49:24.683658+0000 mon.a (mon.0) 362 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[65864]: audit 2023-12-16T07:49:24.686850+0000 mon.a (mon.0) 363 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:25 smithi007 bash[65864]: audit 2023-12-16T07:49:24.690360+0000 mon.a (mon.0) 364 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-12-16T07:49:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:25 smithi169 bash[51638]: audit 2023-12-16T07:49:24.263219+0000 mgr.y (mgr.25477) 222 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:25 smithi169 bash[51638]: audit 2023-12-16T07:49:24.556860+0000 mon.a (mon.0) 355 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:25 smithi169 bash[51638]: audit 2023-12-16T07:49:24.569701+0000 mon.a (mon.0) 356 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:25 smithi169 bash[51638]: audit 2023-12-16T07:49:24.572307+0000 mon.a (mon.0) 357 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:49:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:25 smithi169 bash[51638]: audit 2023-12-16T07:49:24.573860+0000 mon.a (mon.0) 358 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:49:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:25 smithi169 bash[51638]: audit 2023-12-16T07:49:24.582812+0000 mon.a (mon.0) 359 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:25 smithi169 bash[51638]: audit 2023-12-16T07:49:24.673745+0000 mon.a (mon.0) 360 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:49:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:25 smithi169 bash[51638]: audit 2023-12-16T07:49:24.678576+0000 mon.a (mon.0) 361 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:25 smithi169 bash[51638]: audit 2023-12-16T07:49:24.683658+0000 mon.a (mon.0) 362 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:25 smithi169 bash[51638]: audit 2023-12-16T07:49:24.686850+0000 mon.a (mon.0) 363 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:25 smithi169 bash[51638]: audit 2023-12-16T07:49:24.690360+0000 mon.a (mon.0) 364 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-12-16T07:49:26.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:26 smithi007 bash[65864]: audit 2023-12-16T07:49:24.690720+0000 mgr.y (mgr.25477) 223 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-12-16T07:49:26.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:26 smithi007 bash[65864]: cephadm 2023-12-16T07:49:24.692826+0000 mgr.y (mgr.25477) 224 : cephadm 1 Upgrade: osd.3 is safe to restart 2023-12-16T07:49:26.551 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:26 smithi007 bash[65864]: cluster 2023-12-16T07:49:25.205576+0000 mgr.y (mgr.25477) 225 : cluster 0 pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:49:26.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:26 smithi007 bash[64479]: audit 2023-12-16T07:49:24.690720+0000 mgr.y (mgr.25477) 223 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-12-16T07:49:26.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:26 smithi007 bash[64479]: cephadm 2023-12-16T07:49:24.692826+0000 mgr.y (mgr.25477) 224 : cephadm 1 Upgrade: osd.3 is safe to restart 2023-12-16T07:49:26.551 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:26 smithi007 bash[64479]: cluster 2023-12-16T07:49:25.205576+0000 mgr.y (mgr.25477) 225 : cluster 0 pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:49:26.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:26 smithi169 bash[51638]: audit 2023-12-16T07:49:24.690720+0000 mgr.y (mgr.25477) 223 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2023-12-16T07:49:26.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:26 smithi169 bash[51638]: cephadm 2023-12-16T07:49:24.692826+0000 mgr.y (mgr.25477) 224 : cephadm 1 Upgrade: osd.3 is safe to restart 2023-12-16T07:49:26.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:26 smithi169 bash[51638]: cluster 2023-12-16T07:49:25.205576+0000 mgr.y (mgr.25477) 225 : cluster 0 pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:49:28.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:28 smithi169 bash[51638]: cluster 2023-12-16T07:49:27.206465+0000 mgr.y (mgr.25477) 226 : cluster 0 pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2023-12-16T07:49:28.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:28 smithi169 bash[51638]: cephadm 2023-12-16T07:49:27.415315+0000 mgr.y (mgr.25477) 227 : cephadm 1 Upgrade: Updating osd.3 2023-12-16T07:49:28.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:28 smithi169 bash[51638]: audit 2023-12-16T07:49:27.422051+0000 mon.a (mon.0) 365 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:28.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:28 smithi169 bash[51638]: audit 2023-12-16T07:49:27.426301+0000 mon.a (mon.0) 366 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-16T07:49:28.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:28 smithi169 bash[51638]: audit 2023-12-16T07:49:27.427590+0000 mon.a (mon.0) 367 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:49:28.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:28 smithi169 bash[51638]: cephadm 2023-12-16T07:49:27.432481+0000 mgr.y (mgr.25477) 228 : cephadm 1 Deploying daemon osd.3 on smithi007 2023-12-16T07:49:28.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:28 smithi007 bash[65864]: cluster 2023-12-16T07:49:27.206465+0000 mgr.y (mgr.25477) 226 : cluster 0 pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2023-12-16T07:49:28.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:28 smithi007 bash[65864]: cephadm 2023-12-16T07:49:27.415315+0000 mgr.y (mgr.25477) 227 : cephadm 1 Upgrade: Updating osd.3 2023-12-16T07:49:28.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:28 smithi007 bash[65864]: audit 2023-12-16T07:49:27.422051+0000 mon.a (mon.0) 365 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:28.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:28 smithi007 bash[65864]: audit 2023-12-16T07:49:27.426301+0000 mon.a (mon.0) 366 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-16T07:49:28.736 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:28 smithi007 bash[65864]: audit 2023-12-16T07:49:27.427590+0000 mon.a (mon.0) 367 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:49:28.736 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:28 smithi007 bash[65864]: cephadm 2023-12-16T07:49:27.432481+0000 mgr.y (mgr.25477) 228 : cephadm 1 Deploying daemon osd.3 on smithi007 2023-12-16T07:49:28.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:28 smithi007 bash[64479]: cluster 2023-12-16T07:49:27.206465+0000 mgr.y (mgr.25477) 226 : cluster 0 pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2023-12-16T07:49:28.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:28 smithi007 bash[64479]: cephadm 2023-12-16T07:49:27.415315+0000 mgr.y (mgr.25477) 227 : cephadm 1 Upgrade: Updating osd.3 2023-12-16T07:49:28.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:28 smithi007 bash[64479]: audit 2023-12-16T07:49:27.422051+0000 mon.a (mon.0) 365 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:28.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:28 smithi007 bash[64479]: audit 2023-12-16T07:49:27.426301+0000 mon.a (mon.0) 366 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-12-16T07:49:28.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:28 smithi007 bash[64479]: audit 2023-12-16T07:49:27.427590+0000 mon.a (mon.0) 367 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:49:28.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:28 smithi007 bash[64479]: cephadm 2023-12-16T07:49:27.432481+0000 mgr.y (mgr.25477) 228 : cephadm 1 Deploying daemon osd.3 on smithi007 2023-12-16T07:49:30.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:29 smithi169 bash[51638]: cluster 2023-12-16T07:49:29.208065+0000 mgr.y (mgr.25477) 229 : cluster 0 pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:49:30.250 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:29 smithi007 bash[65864]: cluster 2023-12-16T07:49:29.208065+0000 mgr.y (mgr.25477) 229 : cluster 0 pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:49:30.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:29 smithi007 bash[64479]: cluster 2023-12-16T07:49:29.208065+0000 mgr.y (mgr.25477) 229 : cluster 0 pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:49:30.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:49:30 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:49:30] "GET /metrics HTTP/1.1" 200 38334 "" "Prometheus/2.33.4" 2023-12-16T07:49:32.268 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:32 smithi007 systemd[1]: Stopping Ceph osd.3 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:49:32.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:32 smithi007 bash[65864]: cluster 2023-12-16T07:49:31.209186+0000 mgr.y (mgr.25477) 230 : cluster 0 pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:49:32.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:32 smithi007 bash[64479]: cluster 2023-12-16T07:49:31.209186+0000 mgr.y (mgr.25477) 230 : cluster 0 pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:49:32.568 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:32 smithi007 bash[42688]: debug 2023-12-16T07:49:32.327+0000 7f0695b18700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:49:32.568 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:32 smithi007 bash[42688]: debug 2023-12-16T07:49:32.327+0000 7f0695b18700 -1 osd.3 118 *** Got signal Terminated *** 2023-12-16T07:49:32.568 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:32 smithi007 bash[42688]: debug 2023-12-16T07:49:32.327+0000 7f0695b18700 -1 osd.3 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:49:32.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:32 smithi169 bash[51638]: cluster 2023-12-16T07:49:31.209186+0000 mgr.y (mgr.25477) 230 : cluster 0 pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:49:33.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:33 smithi007 bash[64479]: cluster 2023-12-16T07:49:32.332817+0000 mon.a (mon.0) 368 : cluster 1 osd.3 marked itself down and dead 2023-12-16T07:49:33.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:33 smithi007 bash[65864]: cluster 2023-12-16T07:49:32.332817+0000 mon.a (mon.0) 368 : cluster 1 osd.3 marked itself down and dead 2023-12-16T07:49:33.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:33 smithi169 bash[51638]: cluster 2023-12-16T07:49:32.332817+0000 mon.a (mon.0) 368 : cluster 1 osd.3 marked itself down and dead 2023-12-16T07:49:34.551 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[74680]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-3 2023-12-16T07:49:34.552 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[74754]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-3 2023-12-16T07:49:34.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[64479]: cluster 2023-12-16T07:49:33.209951+0000 mgr.y (mgr.25477) 231 : cluster 0 pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:49:34.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[64479]: cluster 2023-12-16T07:49:33.267496+0000 mon.a (mon.0) 369 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:49:34.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[64479]: cluster 2023-12-16T07:49:33.282713+0000 mon.a (mon.0) 370 : cluster 0 osdmap e119: 8 total, 7 up, 8 in 2023-12-16T07:49:34.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[64479]: cluster 2023-12-16T07:49:33.282891+0000 osd.7 (osd.7) 16 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[64479]: cluster 2023-12-16T07:49:33.284088+0000 osd.6 (osd.6) 22 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[64479]: cluster 2023-12-16T07:49:33.284969+0000 osd.5 (osd.5) 19 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[64479]: cluster 2023-12-16T07:49:33.286441+0000 osd.7 (osd.7) 17 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.552 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[64479]: cluster 2023-12-16T07:49:33.289835+0000 osd.7 (osd.7) 18 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[64479]: cluster 2023-12-16T07:49:33.291810+0000 osd.5 (osd.5) 20 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.553 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[64479]: cluster 2023-12-16T07:49:33.293065+0000 osd.7 (osd.7) 19 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[65864]: cluster 2023-12-16T07:49:33.209951+0000 mgr.y (mgr.25477) 231 : cluster 0 pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:49:34.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[65864]: cluster 2023-12-16T07:49:33.267496+0000 mon.a (mon.0) 369 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:49:34.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[65864]: cluster 2023-12-16T07:49:33.282713+0000 mon.a (mon.0) 370 : cluster 0 osdmap e119: 8 total, 7 up, 8 in 2023-12-16T07:49:34.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[65864]: cluster 2023-12-16T07:49:33.282891+0000 osd.7 (osd.7) 16 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[65864]: cluster 2023-12-16T07:49:33.284088+0000 osd.6 (osd.6) 22 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[65864]: cluster 2023-12-16T07:49:33.284969+0000 osd.5 (osd.5) 19 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[65864]: cluster 2023-12-16T07:49:33.286441+0000 osd.7 (osd.7) 17 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[65864]: cluster 2023-12-16T07:49:33.289835+0000 osd.7 (osd.7) 18 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[65864]: cluster 2023-12-16T07:49:33.291810+0000 osd.5 (osd.5) 20 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.553 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:34 smithi007 bash[65864]: cluster 2023-12-16T07:49:33.293065+0000 osd.7 (osd.7) 19 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:34 smithi169 bash[51638]: cluster 2023-12-16T07:49:33.209951+0000 mgr.y (mgr.25477) 231 : cluster 0 pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 161 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:49:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:34 smithi169 bash[51638]: cluster 2023-12-16T07:49:33.267496+0000 mon.a (mon.0) 369 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:49:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:34 smithi169 bash[51638]: cluster 2023-12-16T07:49:33.282713+0000 mon.a (mon.0) 370 : cluster 0 osdmap e119: 8 total, 7 up, 8 in 2023-12-16T07:49:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:34 smithi169 bash[51638]: cluster 2023-12-16T07:49:33.282891+0000 osd.7 (osd.7) 16 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:34 smithi169 bash[51638]: cluster 2023-12-16T07:49:33.284088+0000 osd.6 (osd.6) 22 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:34 smithi169 bash[51638]: cluster 2023-12-16T07:49:33.284969+0000 osd.5 (osd.5) 19 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:34 smithi169 bash[51638]: cluster 2023-12-16T07:49:33.286441+0000 osd.7 (osd.7) 17 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:34 smithi169 bash[51638]: cluster 2023-12-16T07:49:33.289835+0000 osd.7 (osd.7) 18 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:34 smithi169 bash[51638]: cluster 2023-12-16T07:49:33.291810+0000 osd.5 (osd.5) 20 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:34.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:34 smithi169 bash[51638]: cluster 2023-12-16T07:49:33.293065+0000 osd.7 (osd.7) 19 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:35.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[64479]: cluster 2023-12-16T07:49:33.285858+0000 osd.4 (osd.4) 19 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:35.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[64479]: cluster 2023-12-16T07:49:33.292190+0000 osd.4 (osd.4) 20 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:35.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[64479]: audit 2023-12-16T07:49:34.271412+0000 mgr.y (mgr.25477) 232 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:35.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[64479]: cluster 2023-12-16T07:49:34.282982+0000 osd.6 (osd.6) 23 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:35.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[64479]: cluster 2023-12-16T07:49:34.284199+0000 mon.a (mon.0) 371 : cluster 0 osdmap e120: 8 total, 7 up, 8 in 2023-12-16T07:49:35.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[64479]: cluster 2023-12-16T07:49:34.286572+0000 osd.6 (osd.6) 24 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:35.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[64479]: cluster 2023-12-16T07:49:34.287569+0000 osd.5 (osd.5) 21 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:35.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[64479]: cluster 2023-12-16T07:49:34.287570+0000 osd.7 (osd.7) 20 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:35.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[65864]: cluster 2023-12-16T07:49:33.285858+0000 osd.4 (osd.4) 19 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:35.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[65864]: cluster 2023-12-16T07:49:33.292190+0000 osd.4 (osd.4) 20 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:35.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[65864]: audit 2023-12-16T07:49:34.271412+0000 mgr.y (mgr.25477) 232 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:35.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[65864]: cluster 2023-12-16T07:49:34.282982+0000 osd.6 (osd.6) 23 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:35.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[65864]: cluster 2023-12-16T07:49:34.284199+0000 mon.a (mon.0) 371 : cluster 0 osdmap e120: 8 total, 7 up, 8 in 2023-12-16T07:49:35.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[65864]: cluster 2023-12-16T07:49:34.286572+0000 osd.6 (osd.6) 24 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:35.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[65864]: cluster 2023-12-16T07:49:34.287569+0000 osd.5 (osd.5) 21 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:35.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:35 smithi007 bash[65864]: cluster 2023-12-16T07:49:34.287570+0000 osd.7 (osd.7) 20 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:35 smithi169 bash[51638]: cluster 2023-12-16T07:49:33.285858+0000 osd.4 (osd.4) 19 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:35 smithi169 bash[51638]: cluster 2023-12-16T07:49:33.292190+0000 osd.4 (osd.4) 20 : cluster 3 failed to encode map e119 with expected crc 2023-12-16T07:49:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:35 smithi169 bash[51638]: audit 2023-12-16T07:49:34.271412+0000 mgr.y (mgr.25477) 232 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:35 smithi169 bash[51638]: cluster 2023-12-16T07:49:34.282982+0000 osd.6 (osd.6) 23 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:35 smithi169 bash[51638]: cluster 2023-12-16T07:49:34.284199+0000 mon.a (mon.0) 371 : cluster 0 osdmap e120: 8 total, 7 up, 8 in 2023-12-16T07:49:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:35 smithi169 bash[51638]: cluster 2023-12-16T07:49:34.286572+0000 osd.6 (osd.6) 24 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:35 smithi169 bash[51638]: cluster 2023-12-16T07:49:34.287569+0000 osd.5 (osd.5) 21 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:35.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:35 smithi169 bash[51638]: cluster 2023-12-16T07:49:34.287570+0000 osd.7 (osd.7) 20 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:36.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:36 smithi007 bash[64479]: cluster 2023-12-16T07:49:34.283129+0000 osd.4 (osd.4) 21 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:36.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:36 smithi007 bash[64479]: cluster 2023-12-16T07:49:35.211344+0000 mgr.y (mgr.25477) 233 : cluster 0 pgmap v131: 161 pgs: 7 active+undersized, 103 active+clean, 6 stale+active+clean, 38 peering, 5 active+undersized+degraded, 1 activating+undersized+degraded, 1 activating+undersized; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 24/630 objects degraded (3.810%) 2023-12-16T07:49:36.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:36 smithi007 bash[64479]: cluster 2023-12-16T07:49:35.279895+0000 mon.a (mon.0) 372 : cluster 3 Health check failed: Reduced data availability: 5 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2023-12-16T07:49:36.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:36 smithi007 bash[64479]: cluster 2023-12-16T07:49:35.279932+0000 mon.a (mon.0) 373 : cluster 3 Health check failed: Degraded data redundancy: 24/630 objects degraded (3.810%), 6 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:36.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:36 smithi007 bash[65864]: cluster 2023-12-16T07:49:34.283129+0000 osd.4 (osd.4) 21 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:36.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:36 smithi007 bash[65864]: cluster 2023-12-16T07:49:35.211344+0000 mgr.y (mgr.25477) 233 : cluster 0 pgmap v131: 161 pgs: 7 active+undersized, 103 active+clean, 6 stale+active+clean, 38 peering, 5 active+undersized+degraded, 1 activating+undersized+degraded, 1 activating+undersized; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 24/630 objects degraded (3.810%) 2023-12-16T07:49:36.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:36 smithi007 bash[65864]: cluster 2023-12-16T07:49:35.279895+0000 mon.a (mon.0) 372 : cluster 3 Health check failed: Reduced data availability: 5 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2023-12-16T07:49:36.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:36 smithi007 bash[65864]: cluster 2023-12-16T07:49:35.279932+0000 mon.a (mon.0) 373 : cluster 3 Health check failed: Degraded data redundancy: 24/630 objects degraded (3.810%), 6 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:36.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:36 smithi169 bash[51638]: cluster 2023-12-16T07:49:34.283129+0000 osd.4 (osd.4) 21 : cluster 3 failed to encode map e120 with expected crc 2023-12-16T07:49:36.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:36 smithi169 bash[51638]: cluster 2023-12-16T07:49:35.211344+0000 mgr.y (mgr.25477) 233 : cluster 0 pgmap v131: 161 pgs: 7 active+undersized, 103 active+clean, 6 stale+active+clean, 38 peering, 5 active+undersized+degraded, 1 activating+undersized+degraded, 1 activating+undersized; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 24/630 objects degraded (3.810%) 2023-12-16T07:49:36.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:36 smithi169 bash[51638]: cluster 2023-12-16T07:49:35.279895+0000 mon.a (mon.0) 372 : cluster 3 Health check failed: Reduced data availability: 5 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2023-12-16T07:49:36.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:36 smithi169 bash[51638]: cluster 2023-12-16T07:49:35.279932+0000 mon.a (mon.0) 373 : cluster 3 Health check failed: Degraded data redundancy: 24/630 objects degraded (3.810%), 6 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:36.859 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:36 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.3.service: Succeeded. 2023-12-16T07:49:36.860 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:36 smithi007 systemd[1]: Stopped Ceph osd.3 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:49:36.860 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:36 smithi007 systemd[1]: Started Ceph osd.3 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:49:38.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:37 smithi007 bash[65864]: audit 2023-12-16T07:49:36.911921+0000 mon.a (mon.0) 374 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:38.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:37 smithi007 bash[65864]: audit 2023-12-16T07:49:36.919863+0000 mon.a (mon.0) 375 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:38.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:37 smithi007 bash[65864]: cluster 2023-12-16T07:49:37.212675+0000 mgr.y (mgr.25477) 234 : cluster 0 pgmap v132: 161 pgs: 16 active+undersized, 95 active+clean, 38 peering, 10 active+undersized+degraded, 1 activating+undersized+degraded, 1 activating+undersized; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 41/630 objects degraded (6.508%) 2023-12-16T07:49:38.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:37 smithi007 bash[64479]: audit 2023-12-16T07:49:36.911921+0000 mon.a (mon.0) 374 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:38.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:37 smithi007 bash[64479]: audit 2023-12-16T07:49:36.919863+0000 mon.a (mon.0) 375 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:38.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:37 smithi007 bash[64479]: cluster 2023-12-16T07:49:37.212675+0000 mgr.y (mgr.25477) 234 : cluster 0 pgmap v132: 161 pgs: 16 active+undersized, 95 active+clean, 38 peering, 10 active+undersized+degraded, 1 activating+undersized+degraded, 1 activating+undersized; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 41/630 objects degraded (6.508%) 2023-12-16T07:49:38.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:37 smithi169 bash[51638]: audit 2023-12-16T07:49:36.911921+0000 mon.a (mon.0) 374 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:38.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:37 smithi169 bash[51638]: audit 2023-12-16T07:49:36.919863+0000 mon.a (mon.0) 375 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:38.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:37 smithi169 bash[51638]: cluster 2023-12-16T07:49:37.212675+0000 mgr.y (mgr.25477) 234 : cluster 0 pgmap v132: 161 pgs: 16 active+undersized, 95 active+clean, 38 peering, 10 active+undersized+degraded, 1 activating+undersized+degraded, 1 activating+undersized; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 41/630 objects degraded (6.508%) 2023-12-16T07:49:38.817 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:38 smithi007 bash[74922]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-16T07:49:38.818 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:38 smithi007 bash[74922]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2023-12-16T07:49:38.818 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:38 smithi007 bash[74922]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2023-12-16T07:49:38.818 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:38 smithi007 bash[74922]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2023-12-16T07:49:38.818 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:38 smithi007 bash[74922]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2023-12-16T07:49:38.818 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:38 smithi007 bash[74922]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-16T07:49:38.818 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:38 smithi007 bash[74922]: --> ceph-volume raw activate successful for osd ID: 3 2023-12-16T07:49:39.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:39 smithi007 bash[64479]: audit 2023-12-16T07:49:38.209198+0000 mon.a (mon.0) 376 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:39.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:39 smithi007 bash[64479]: audit 2023-12-16T07:49:38.210894+0000 mon.a (mon.0) 377 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:49:39.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:39 smithi007 bash[65864]: audit 2023-12-16T07:49:38.209198+0000 mon.a (mon.0) 376 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:39.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:39 smithi007 bash[65864]: audit 2023-12-16T07:49:38.210894+0000 mon.a (mon.0) 377 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:49:39.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:39 smithi169 bash[51638]: audit 2023-12-16T07:49:38.209198+0000 mon.a (mon.0) 376 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:39.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:39 smithi169 bash[51638]: audit 2023-12-16T07:49:38.210894+0000 mon.a (mon.0) 377 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:49:40.317 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:40 smithi007 bash[75209]: debug 2023-12-16T07:49:40.159+0000 7fb6f05e2700 1 -- 172.21.15.7:0/2104730108 <== mon.1 v2:172.21.15.7:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557d0d2b76c0 con 0x557d0d33cc00 2023-12-16T07:49:40.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:40 smithi007 bash[64479]: cluster 2023-12-16T07:49:39.213978+0000 mgr.y (mgr.25477) 235 : cluster 0 pgmap v133: 161 pgs: 39 active+undersized, 95 active+clean, 25 active+undersized+degraded, 1 activating+undersized+degraded, 1 activating+undersized; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:49:40.318 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:49:40 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:49:40] "GET /metrics HTTP/1.1" 200 38401 "" "Prometheus/2.33.4" 2023-12-16T07:49:40.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:40 smithi007 bash[65864]: cluster 2023-12-16T07:49:39.213978+0000 mgr.y (mgr.25477) 235 : cluster 0 pgmap v133: 161 pgs: 39 active+undersized, 95 active+clean, 25 active+undersized+degraded, 1 activating+undersized+degraded, 1 activating+undersized; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:49:40.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:40 smithi169 bash[51638]: cluster 2023-12-16T07:49:39.213978+0000 mgr.y (mgr.25477) 235 : cluster 0 pgmap v133: 161 pgs: 39 active+undersized, 95 active+clean, 25 active+undersized+degraded, 1 activating+undersized+degraded, 1 activating+undersized; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:49:41.067 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:40 smithi007 bash[75209]: debug 2023-12-16T07:49:40.699+0000 7fb6f9768640 -1 Falling back to public interface 2023-12-16T07:49:41.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:41 smithi007 bash[64479]: cluster 2023-12-16T07:49:40.207686+0000 mon.a (mon.0) 378 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 11 pgs peering) 2023-12-16T07:49:41.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:41 smithi007 bash[65864]: cluster 2023-12-16T07:49:40.207686+0000 mon.a (mon.0) 378 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 11 pgs peering) 2023-12-16T07:49:41.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:41 smithi169 bash[51638]: cluster 2023-12-16T07:49:40.207686+0000 mon.a (mon.0) 378 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 11 pgs peering) 2023-12-16T07:49:42.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:42 smithi007 bash[64479]: cluster 2023-12-16T07:49:41.215513+0000 mgr.y (mgr.25477) 236 : cluster 0 pgmap v134: 161 pgs: 39 active+undersized, 95 active+clean, 27 active+undersized+degraded; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:49:42.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:42 smithi007 bash[64479]: cluster 2023-12-16T07:49:41.267664+0000 mon.a (mon.0) 379 : cluster 3 Health check update: Degraded data redundancy: 82/630 objects degraded (13.016%), 27 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:42.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:42 smithi007 bash[65864]: cluster 2023-12-16T07:49:41.215513+0000 mgr.y (mgr.25477) 236 : cluster 0 pgmap v134: 161 pgs: 39 active+undersized, 95 active+clean, 27 active+undersized+degraded; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:49:42.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:42 smithi007 bash[65864]: cluster 2023-12-16T07:49:41.267664+0000 mon.a (mon.0) 379 : cluster 3 Health check update: Degraded data redundancy: 82/630 objects degraded (13.016%), 27 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:42.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:42 smithi169 bash[51638]: cluster 2023-12-16T07:49:41.215513+0000 mgr.y (mgr.25477) 236 : cluster 0 pgmap v134: 161 pgs: 39 active+undersized, 95 active+clean, 27 active+undersized+degraded; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:49:42.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:42 smithi169 bash[51638]: cluster 2023-12-16T07:49:41.267664+0000 mon.a (mon.0) 379 : cluster 3 Health check update: Degraded data redundancy: 82/630 objects degraded (13.016%), 27 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:44.067 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:43 smithi007 bash[75209]: debug 2023-12-16T07:49:43.715+0000 7fb6f9768640 -1 osd.3 0 read_superblock omap replica is missing. 2023-12-16T07:49:44.068 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:43 smithi007 bash[75209]: debug 2023-12-16T07:49:43.767+0000 7fb6f9768640 -1 osd.3 118 log_to_monitors true 2023-12-16T07:49:44.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:44 smithi007 bash[65864]: cluster 2023-12-16T07:49:43.216409+0000 mgr.y (mgr.25477) 237 : cluster 0 pgmap v135: 161 pgs: 39 active+undersized, 95 active+clean, 27 active+undersized+degraded; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:49:44.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:44 smithi007 bash[65864]: audit 2023-12-16T07:49:43.516127+0000 mon.a (mon.0) 380 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:44.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:44 smithi007 bash[65864]: audit 2023-12-16T07:49:43.528132+0000 mon.a (mon.0) 381 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:44.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:44 smithi007 bash[65864]: audit 2023-12-16T07:49:43.778693+0000 mon.c (mon.1) 11 : audit 1 from='osd.3 [v2:172.21.15.7:6826/1836732346,v1:172.21.15.7:6827/1836732346]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T07:49:44.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:44 smithi007 bash[65864]: audit 2023-12-16T07:49:43.779373+0000 mon.a (mon.0) 382 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T07:49:44.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:44 smithi007 bash[64479]: cluster 2023-12-16T07:49:43.216409+0000 mgr.y (mgr.25477) 237 : cluster 0 pgmap v135: 161 pgs: 39 active+undersized, 95 active+clean, 27 active+undersized+degraded; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:49:44.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:44 smithi007 bash[64479]: audit 2023-12-16T07:49:43.516127+0000 mon.a (mon.0) 380 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:44.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:44 smithi007 bash[64479]: audit 2023-12-16T07:49:43.528132+0000 mon.a (mon.0) 381 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:44.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:44 smithi007 bash[64479]: audit 2023-12-16T07:49:43.778693+0000 mon.c (mon.1) 11 : audit 1 from='osd.3 [v2:172.21.15.7:6826/1836732346,v1:172.21.15.7:6827/1836732346]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T07:49:44.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:44 smithi007 bash[64479]: audit 2023-12-16T07:49:43.779373+0000 mon.a (mon.0) 382 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T07:49:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:44 smithi169 bash[51638]: cluster 2023-12-16T07:49:43.216409+0000 mgr.y (mgr.25477) 237 : cluster 0 pgmap v135: 161 pgs: 39 active+undersized, 95 active+clean, 27 active+undersized+degraded; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:49:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:44 smithi169 bash[51638]: audit 2023-12-16T07:49:43.516127+0000 mon.a (mon.0) 380 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:44 smithi169 bash[51638]: audit 2023-12-16T07:49:43.528132+0000 mon.a (mon.0) 381 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:44 smithi169 bash[51638]: audit 2023-12-16T07:49:43.778693+0000 mon.c (mon.1) 11 : audit 1 from='osd.3 [v2:172.21.15.7:6826/1836732346,v1:172.21.15.7:6827/1836732346]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T07:49:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:44 smithi169 bash[51638]: audit 2023-12-16T07:49:43.779373+0000 mon.a (mon.0) 382 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-12-16T07:49:45.067 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:49:44 smithi007 bash[75209]: debug 2023-12-16T07:49:44.563+0000 7fb6e83d4700 -1 osd.3 118 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:49:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[65864]: audit 2023-12-16T07:49:44.280092+0000 mgr.y (mgr.25477) 238 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[65864]: audit 2023-12-16T07:49:44.525955+0000 mon.a (mon.0) 383 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-16T07:49:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[65864]: cluster 2023-12-16T07:49:44.532007+0000 mon.a (mon.0) 384 : cluster 0 osdmap e121: 8 total, 7 up, 8 in 2023-12-16T07:49:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[65864]: audit 2023-12-16T07:49:44.535196+0000 mon.c (mon.1) 12 : audit 1 from='osd.3 [v2:172.21.15.7:6826/1836732346,v1:172.21.15.7:6827/1836732346]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:49:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[65864]: audit 2023-12-16T07:49:44.535685+0000 mon.a (mon.0) 385 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:49:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[65864]: audit 2023-12-16T07:49:44.622420+0000 mon.a (mon.0) 386 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:45.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[65864]: audit 2023-12-16T07:49:44.632612+0000 mon.a (mon.0) 387 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:45.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.099927+0000 osd.4 (osd.4) 22 : cluster 3 failed to encode map e121 with expected crc 2023-12-16T07:49:45.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.099928+0000 osd.5 (osd.5) 22 : cluster 3 failed to encode map e121 with expected crc 2023-12-16T07:49:45.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.100163+0000 osd.6 (osd.6) 25 : cluster 3 failed to encode map e121 with expected crc 2023-12-16T07:49:45.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.100252+0000 osd.7 (osd.7) 21 : cluster 3 failed to encode map e121 with expected crc 2023-12-16T07:49:45.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[64479]: audit 2023-12-16T07:49:44.280092+0000 mgr.y (mgr.25477) 238 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:45.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[64479]: audit 2023-12-16T07:49:44.525955+0000 mon.a (mon.0) 383 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-16T07:49:45.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[64479]: cluster 2023-12-16T07:49:44.532007+0000 mon.a (mon.0) 384 : cluster 0 osdmap e121: 8 total, 7 up, 8 in 2023-12-16T07:49:45.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[64479]: audit 2023-12-16T07:49:44.535196+0000 mon.c (mon.1) 12 : audit 1 from='osd.3 [v2:172.21.15.7:6826/1836732346,v1:172.21.15.7:6827/1836732346]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:49:45.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[64479]: audit 2023-12-16T07:49:44.535685+0000 mon.a (mon.0) 385 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:49:45.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[64479]: audit 2023-12-16T07:49:44.622420+0000 mon.a (mon.0) 386 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:45.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[64479]: audit 2023-12-16T07:49:44.632612+0000 mon.a (mon.0) 387 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:45.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.099927+0000 osd.4 (osd.4) 22 : cluster 3 failed to encode map e121 with expected crc 2023-12-16T07:49:45.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.099928+0000 osd.5 (osd.5) 22 : cluster 3 failed to encode map e121 with expected crc 2023-12-16T07:49:45.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.100163+0000 osd.6 (osd.6) 25 : cluster 3 failed to encode map e121 with expected crc 2023-12-16T07:49:45.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:45 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.100252+0000 osd.7 (osd.7) 21 : cluster 3 failed to encode map e121 with expected crc 2023-12-16T07:49:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:45 smithi169 bash[51638]: audit 2023-12-16T07:49:44.280092+0000 mgr.y (mgr.25477) 238 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:45 smithi169 bash[51638]: audit 2023-12-16T07:49:44.525955+0000 mon.a (mon.0) 383 : audit 1 from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-12-16T07:49:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:45 smithi169 bash[51638]: cluster 2023-12-16T07:49:44.532007+0000 mon.a (mon.0) 384 : cluster 0 osdmap e121: 8 total, 7 up, 8 in 2023-12-16T07:49:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:45 smithi169 bash[51638]: audit 2023-12-16T07:49:44.535196+0000 mon.c (mon.1) 12 : audit 1 from='osd.3 [v2:172.21.15.7:6826/1836732346,v1:172.21.15.7:6827/1836732346]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:49:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:45 smithi169 bash[51638]: audit 2023-12-16T07:49:44.535685+0000 mon.a (mon.0) 385 : audit 1 from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2023-12-16T07:49:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:45 smithi169 bash[51638]: audit 2023-12-16T07:49:44.622420+0000 mon.a (mon.0) 386 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:45 smithi169 bash[51638]: audit 2023-12-16T07:49:44.632612+0000 mon.a (mon.0) 387 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:45 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.099927+0000 osd.4 (osd.4) 22 : cluster 3 failed to encode map e121 with expected crc 2023-12-16T07:49:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:45 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.099928+0000 osd.5 (osd.5) 22 : cluster 3 failed to encode map e121 with expected crc 2023-12-16T07:49:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:45 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.100163+0000 osd.6 (osd.6) 25 : cluster 3 failed to encode map e121 with expected crc 2023-12-16T07:49:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:45 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.100252+0000 osd.7 (osd.7) 21 : cluster 3 failed to encode map e121 with expected crc 2023-12-16T07:49:46.017 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:49:46.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.217415+0000 mgr.y (mgr.25477) 239 : cluster 0 pgmap v137: 161 pgs: 39 active+undersized, 95 active+clean, 27 active+undersized+degraded; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 82/630 objects degraded (13.016%) 2023-12-16T07:49:46.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.527117+0000 mon.a (mon.0) 388 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:49:46.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.533953+0000 osd.4 (osd.4) 23 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.536537+0000 mon.a (mon.0) 389 : cluster 1 osd.3 [v2:172.21.15.7:6826/1836732346,v1:172.21.15.7:6827/1836732346] boot 2023-12-16T07:49:46.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.536625+0000 mon.a (mon.0) 390 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2023-12-16T07:49:46.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: audit 2023-12-16T07:49:45.537541+0000 mon.a (mon.0) 391 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:49:46.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.540041+0000 osd.4 (osd.4) 24 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.540518+0000 osd.5 (osd.5) 23 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.540869+0000 osd.7 (osd.7) 22 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.543305+0000 osd.4 (osd.4) 25 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.546010+0000 osd.7 (osd.7) 23 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.546719+0000 osd.5 (osd.5) 24 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.549369+0000 osd.4 (osd.4) 26 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.551689+0000 osd.5 (osd.5) 25 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:45.551860+0000 osd.7 (osd.7) 24 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: audit 2023-12-16T07:49:45.998956+0000 mgr.y (mgr.25477) 240 : audit 0 from='client.44341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:46.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: audit 2023-12-16T07:49:46.400324+0000 mgr.y (mgr.25477) 241 : audit 0 from='client.34413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:46.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[65864]: cluster 2023-12-16T07:49:46.537335+0000 mon.a (mon.0) 392 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2023-12-16T07:49:46.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.217415+0000 mgr.y (mgr.25477) 239 : cluster 0 pgmap v137: 161 pgs: 39 active+undersized, 95 active+clean, 27 active+undersized+degraded; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 82/630 objects degraded (13.016%) 2023-12-16T07:49:46.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.527117+0000 mon.a (mon.0) 388 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:49:46.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.533953+0000 osd.4 (osd.4) 23 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.536537+0000 mon.a (mon.0) 389 : cluster 1 osd.3 [v2:172.21.15.7:6826/1836732346,v1:172.21.15.7:6827/1836732346] boot 2023-12-16T07:49:46.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.536625+0000 mon.a (mon.0) 390 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2023-12-16T07:49:46.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: audit 2023-12-16T07:49:45.537541+0000 mon.a (mon.0) 391 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:49:46.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.540041+0000 osd.4 (osd.4) 24 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.540518+0000 osd.5 (osd.5) 23 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.540869+0000 osd.7 (osd.7) 22 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.543305+0000 osd.4 (osd.4) 25 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.546010+0000 osd.7 (osd.7) 23 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.546719+0000 osd.5 (osd.5) 24 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.549369+0000 osd.4 (osd.4) 26 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.551689+0000 osd.5 (osd.5) 25 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:45.551860+0000 osd.7 (osd.7) 24 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: audit 2023-12-16T07:49:45.998956+0000 mgr.y (mgr.25477) 240 : audit 0 from='client.44341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:46.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: audit 2023-12-16T07:49:46.400324+0000 mgr.y (mgr.25477) 241 : audit 0 from='client.34413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:46.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:46 smithi007 bash[64479]: cluster 2023-12-16T07:49:46.537335+0000 mon.a (mon.0) 392 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2023-12-16T07:49:46.828 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:49:46.828 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (6m) 3s ago 13m 16.8M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:49:46.828 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (5m) 2m ago 13m 43.6M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:49:46.828 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (4m) 3s ago 11m 47.7M - 3.5 e1d6a67b021e 8d8009d0c83f 2023-12-16T07:49:46.828 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283,8765 running (4m) 2m ago 19m 433M - 19.0.0-191-g6d933094 a29c258b5d6f 5295ba35c2cd 2023-12-16T07:49:46.828 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (5m) 3s ago 21m 493M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:49:46.828 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (4m) 3s ago 21m 52.7M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 5a266b45f3ce 2023-12-16T07:49:46.829 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (3m) 2m ago 19m 23.6M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1fc9bf864dda 2023-12-16T07:49:46.829 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (3m) 3s ago 19m 42.0M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1dc91c31fc27 2023-12-16T07:49:46.829 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (6m) 3s ago 13m 13.0M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:49:46.829 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (5m) 2m ago 13m 11.3M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:49:46.829 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (103s) 3s ago 18m 52.0M 2782M 19.0.0-191-g6d933094 a29c258b5d6f a00724c6d077 2023-12-16T07:49:46.829 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (71s) 3s ago 18m 51.3M 2782M 19.0.0-191-g6d933094 a29c258b5d6f 5e90fce88079 2023-12-16T07:49:46.829 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (39s) 3s ago 17m 46.9M 2782M 19.0.0-191-g6d933094 a29c258b5d6f 906a30291e88 2023-12-16T07:49:46.829 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (7s) 3s ago 16m - 2782M 19.0.0-191-g6d933094 a29c258b5d6f 8e5e7d6ede0f 2023-12-16T07:49:46.829 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (16m) 2m ago 16m 58.5M 3037M 17.2.0 e1d6a67b021e 0196c30e9d2e 2023-12-16T07:49:46.829 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (15m) 2m ago 15m 55.3M 3037M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:49:46.830 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (15m) 2m ago 15m 55.8M 3037M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:49:46.830 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (14m) 2m ago 14m 56.9M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:49:46.830 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (4m) 2m ago 13m 59.0M - 2.33.4 514e6a882f6e 2652ce1b8540 2023-12-16T07:49:46.830 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (11m) 3s ago 11m 88.1M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:49:46.830 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (11m) 2m ago 11m 87.4M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:49:46.830 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (11m) 3s ago 11m 88.0M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:49:46.830 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (11m) 2m ago 11m 87.0M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:49:46.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.217415+0000 mgr.y (mgr.25477) 239 : cluster 0 pgmap v137: 161 pgs: 39 active+undersized, 95 active+clean, 27 active+undersized+degraded; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 82/630 objects degraded (13.016%) 2023-12-16T07:49:46.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.527117+0000 mon.a (mon.0) 388 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:49:46.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.533953+0000 osd.4 (osd.4) 23 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.536537+0000 mon.a (mon.0) 389 : cluster 1 osd.3 [v2:172.21.15.7:6826/1836732346,v1:172.21.15.7:6827/1836732346] boot 2023-12-16T07:49:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.536625+0000 mon.a (mon.0) 390 : cluster 0 osdmap e122: 8 total, 8 up, 8 in 2023-12-16T07:49:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: audit 2023-12-16T07:49:45.537541+0000 mon.a (mon.0) 391 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-12-16T07:49:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.540041+0000 osd.4 (osd.4) 24 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.540518+0000 osd.5 (osd.5) 23 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.540869+0000 osd.7 (osd.7) 22 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.543305+0000 osd.4 (osd.4) 25 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.546010+0000 osd.7 (osd.7) 23 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.546719+0000 osd.5 (osd.5) 24 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.549369+0000 osd.4 (osd.4) 26 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.551689+0000 osd.5 (osd.5) 25 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:45.551860+0000 osd.7 (osd.7) 24 : cluster 3 failed to encode map e122 with expected crc 2023-12-16T07:49:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: audit 2023-12-16T07:49:45.998956+0000 mgr.y (mgr.25477) 240 : audit 0 from='client.44341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: audit 2023-12-16T07:49:46.400324+0000 mgr.y (mgr.25477) 241 : audit 0 from='client.34413 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:46.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:46 smithi169 bash[51638]: cluster 2023-12-16T07:49:46.537335+0000 mon.a (mon.0) 392 : cluster 0 osdmap e123: 8 total, 8 up, 8 in 2023-12-16T07:49:47.293 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:49:47.293 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:49:47.293 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 3 2023-12-16T07:49:47.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:49:47.294 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:49:47.294 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:49:47.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:49:47.294 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:49:47.294 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2023-12-16T07:49:47.294 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 4 2023-12-16T07:49:47.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:49:47.294 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:49:47.294 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:49:47.294 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:49:47.294 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:49:47.294 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2023-12-16T07:49:47.295 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 9 2023-12-16T07:49:47.295 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:49:47.295 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:49:47.684 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:49:47.684 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:49:47.684 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:49:47.684 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:49:47.684 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2023-12-16T07:49:47.684 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr", 2023-12-16T07:49:47.684 INFO:teuthology.orchestra.run.smithi007.stdout: "mon" 2023-12-16T07:49:47.684 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:49:47.684 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "9/23 daemons upgraded", 2023-12-16T07:49:47.685 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading osd daemons", 2023-12-16T07:49:47.685 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:49:47.685 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:49:47.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: cluster 2023-12-16T07:49:46.538861+0000 osd.7 (osd.7) 25 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: cluster 2023-12-16T07:49:46.539999+0000 osd.6 (osd.6) 26 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: cluster 2023-12-16T07:49:46.541427+0000 osd.5 (osd.5) 26 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: cluster 2023-12-16T07:49:46.541609+0000 osd.4 (osd.4) 27 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: cluster 2023-12-16T07:49:46.544772+0000 osd.4 (osd.4) 28 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: cluster 2023-12-16T07:49:46.546686+0000 osd.6 (osd.6) 27 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: cluster 2023-12-16T07:49:46.546739+0000 osd.5 (osd.5) 27 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: cluster 2023-12-16T07:49:46.548412+0000 osd.7 (osd.7) 26 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: cluster 2023-12-16T07:49:46.548494+0000 osd.4 (osd.4) 29 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: cluster 2023-12-16T07:49:46.549978+0000 osd.5 (osd.5) 28 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: cluster 2023-12-16T07:49:46.551467+0000 osd.7 (osd.7) 27 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: cluster 2023-12-16T07:49:46.553509+0000 osd.5 (osd.5) 29 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: audit 2023-12-16T07:49:46.807355+0000 mgr.y (mgr.25477) 242 : audit 0 from='client.44350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:47.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: cluster 2023-12-16T07:49:47.218380+0000 mgr.y (mgr.25477) 243 : cluster 0 pgmap v140: 161 pgs: 8 peering, 34 active+undersized, 95 active+clean, 24 active+undersized+degraded; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2023-12-16T07:49:47.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[64479]: audit 2023-12-16T07:49:47.290499+0000 mon.a (mon.0) 393 : audit 0 from='client.? 172.21.15.7:0/3623169505' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:47.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: cluster 2023-12-16T07:49:46.538861+0000 osd.7 (osd.7) 25 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: cluster 2023-12-16T07:49:46.539999+0000 osd.6 (osd.6) 26 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: cluster 2023-12-16T07:49:46.541427+0000 osd.5 (osd.5) 26 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: cluster 2023-12-16T07:49:46.541609+0000 osd.4 (osd.4) 27 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: cluster 2023-12-16T07:49:46.544772+0000 osd.4 (osd.4) 28 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: cluster 2023-12-16T07:49:46.546686+0000 osd.6 (osd.6) 27 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: cluster 2023-12-16T07:49:46.546739+0000 osd.5 (osd.5) 27 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: cluster 2023-12-16T07:49:46.548412+0000 osd.7 (osd.7) 26 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: cluster 2023-12-16T07:49:46.548494+0000 osd.4 (osd.4) 29 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: cluster 2023-12-16T07:49:46.549978+0000 osd.5 (osd.5) 28 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: cluster 2023-12-16T07:49:46.551467+0000 osd.7 (osd.7) 27 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: cluster 2023-12-16T07:49:46.553509+0000 osd.5 (osd.5) 29 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: audit 2023-12-16T07:49:46.807355+0000 mgr.y (mgr.25477) 242 : audit 0 from='client.44350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:47.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: cluster 2023-12-16T07:49:47.218380+0000 mgr.y (mgr.25477) 243 : cluster 0 pgmap v140: 161 pgs: 8 peering, 34 active+undersized, 95 active+clean, 24 active+undersized+degraded; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2023-12-16T07:49:47.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:47 smithi007 bash[65864]: audit 2023-12-16T07:49:47.290499+0000 mon.a (mon.0) 393 : audit 0 from='client.? 172.21.15.7:0/3623169505' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:47.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: cluster 2023-12-16T07:49:46.538861+0000 osd.7 (osd.7) 25 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: cluster 2023-12-16T07:49:46.539999+0000 osd.6 (osd.6) 26 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: cluster 2023-12-16T07:49:46.541427+0000 osd.5 (osd.5) 26 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: cluster 2023-12-16T07:49:46.541609+0000 osd.4 (osd.4) 27 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: cluster 2023-12-16T07:49:46.544772+0000 osd.4 (osd.4) 28 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: cluster 2023-12-16T07:49:46.546686+0000 osd.6 (osd.6) 27 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: cluster 2023-12-16T07:49:46.546739+0000 osd.5 (osd.5) 27 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: cluster 2023-12-16T07:49:46.548412+0000 osd.7 (osd.7) 26 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: cluster 2023-12-16T07:49:46.548494+0000 osd.4 (osd.4) 29 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: cluster 2023-12-16T07:49:46.549978+0000 osd.5 (osd.5) 28 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: cluster 2023-12-16T07:49:46.551467+0000 osd.7 (osd.7) 27 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: cluster 2023-12-16T07:49:46.553509+0000 osd.5 (osd.5) 29 : cluster 3 failed to encode map e123 with expected crc 2023-12-16T07:49:47.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: audit 2023-12-16T07:49:46.807355+0000 mgr.y (mgr.25477) 242 : audit 0 from='client.44350 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:47.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: cluster 2023-12-16T07:49:47.218380+0000 mgr.y (mgr.25477) 243 : cluster 0 pgmap v140: 161 pgs: 8 peering, 34 active+undersized, 95 active+clean, 24 active+undersized+degraded; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 73/630 objects degraded (11.587%) 2023-12-16T07:49:47.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:47 smithi169 bash[51638]: audit 2023-12-16T07:49:47.290499+0000 mon.a (mon.0) 393 : audit 0 from='client.? 172.21.15.7:0/3623169505' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:48.125 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_WARN Degraded data redundancy: 73/630 objects degraded (11.587%), 24 pgs degraded 2023-12-16T07:49:48.125 INFO:teuthology.orchestra.run.smithi007.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 73/630 objects degraded (11.587%), 24 pgs degraded 2023-12-16T07:49:48.126 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.1 is active+undersized+degraded, acting [2,0] 2023-12-16T07:49:48.126 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.e is active+undersized+degraded, acting [2,7] 2023-12-16T07:49:48.126 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.12 is active+undersized+degraded, acting [5,7] 2023-12-16T07:49:48.126 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.13 is active+undersized+degraded, acting [0,4] 2023-12-16T07:49:48.126 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.1b is active+undersized+degraded, acting [7,6] 2023-12-16T07:49:48.126 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.1e is active+undersized+degraded, acting [0,5] 2023-12-16T07:49:48.126 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.1 is active+undersized+degraded, acting [0,4] 2023-12-16T07:49:48.126 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.2 is active+undersized+degraded, acting [5,6] 2023-12-16T07:49:48.126 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.5 is active+undersized+degraded, acting [5,2] 2023-12-16T07:49:48.126 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.7 is active+undersized+degraded, acting [7,0] 2023-12-16T07:49:48.126 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.8 is active+undersized+degraded, acting [1,7] 2023-12-16T07:49:48.126 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.b is active+undersized+degraded, acting [0,4] 2023-12-16T07:49:48.127 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.c is active+undersized+degraded, acting [5,6] 2023-12-16T07:49:48.127 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.12 is active+undersized+degraded, acting [0,7] 2023-12-16T07:49:48.127 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.15 is active+undersized+degraded, acting [7,4] 2023-12-16T07:49:48.127 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.17 is active+undersized+degraded, acting [0,5] 2023-12-16T07:49:48.127 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.18 is active+undersized+degraded, acting [0,1] 2023-12-16T07:49:48.127 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.1e is active+undersized+degraded, acting [6,2] 2023-12-16T07:49:48.127 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.0 is active+undersized+degraded, acting [7,0] 2023-12-16T07:49:48.127 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.9 is active+undersized+degraded, acting [4,1] 2023-12-16T07:49:48.127 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.14 is active+undersized+degraded, acting [1,7] 2023-12-16T07:49:48.127 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.15 is active+undersized+degraded, acting [5,7] 2023-12-16T07:49:48.127 INFO:teuthology.orchestra.run.smithi007.stdout: pg 5.1c is active+undersized+degraded, acting [4,2] 2023-12-16T07:49:48.127 INFO:teuthology.orchestra.run.smithi007.stdout: pg 6.c is active+undersized+degraded, acting [6,5] 2023-12-16T07:49:48.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:48 smithi007 bash[65864]: cluster 2023-12-16T07:49:47.551269+0000 mon.a (mon.0) 394 : cluster 3 Health check update: Degraded data redundancy: 73/630 objects degraded (11.587%), 24 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:48.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:48 smithi007 bash[65864]: audit 2023-12-16T07:49:47.680364+0000 mgr.y (mgr.25477) 244 : audit 0 from='client.34431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:48.822 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:48 smithi007 bash[65864]: audit 2023-12-16T07:49:48.121420+0000 mon.a (mon.0) 395 : audit 0 from='client.? 172.21.15.7:0/2095993055' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:49:48.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:48 smithi007 bash[64479]: cluster 2023-12-16T07:49:47.551269+0000 mon.a (mon.0) 394 : cluster 3 Health check update: Degraded data redundancy: 73/630 objects degraded (11.587%), 24 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:48.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:48 smithi007 bash[64479]: audit 2023-12-16T07:49:47.680364+0000 mgr.y (mgr.25477) 244 : audit 0 from='client.34431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:48.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:48 smithi007 bash[64479]: audit 2023-12-16T07:49:48.121420+0000 mon.a (mon.0) 395 : audit 0 from='client.? 172.21.15.7:0/2095993055' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:49:48.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:48 smithi169 bash[51638]: cluster 2023-12-16T07:49:47.551269+0000 mon.a (mon.0) 394 : cluster 3 Health check update: Degraded data redundancy: 73/630 objects degraded (11.587%), 24 pgs degraded (PG_DEGRADED) 2023-12-16T07:49:48.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:48 smithi169 bash[51638]: audit 2023-12-16T07:49:47.680364+0000 mgr.y (mgr.25477) 244 : audit 0 from='client.34431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:49:48.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:48 smithi169 bash[51638]: audit 2023-12-16T07:49:48.121420+0000 mon.a (mon.0) 395 : audit 0 from='client.? 172.21.15.7:0/2095993055' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:49:50.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:49 smithi007 bash[64479]: cluster 2023-12-16T07:49:49.219996+0000 mgr.y (mgr.25477) 245 : cluster 0 pgmap v141: 161 pgs: 8 peering, 20 active+undersized, 118 active+clean, 15 active+undersized+degraded; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 50/630 objects degraded (7.937%) 2023-12-16T07:49:50.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:49 smithi007 bash[65864]: cluster 2023-12-16T07:49:49.219996+0000 mgr.y (mgr.25477) 245 : cluster 0 pgmap v141: 161 pgs: 8 peering, 20 active+undersized, 118 active+clean, 15 active+undersized+degraded; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 50/630 objects degraded (7.937%) 2023-12-16T07:49:50.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:49 smithi169 bash[51638]: cluster 2023-12-16T07:49:49.219996+0000 mgr.y (mgr.25477) 245 : cluster 0 pgmap v141: 161 pgs: 8 peering, 20 active+undersized, 118 active+clean, 15 active+undersized+degraded; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 50/630 objects degraded (7.937%) 2023-12-16T07:49:50.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:49:50 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:49:50] "GET /metrics HTTP/1.1" 200 38401 "" "Prometheus/2.33.4" 2023-12-16T07:49:51.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:51 smithi007 bash[64479]: cluster 2023-12-16T07:49:51.273153+0000 mon.a (mon.0) 396 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50/630 objects degraded (7.937%), 15 pgs degraded) 2023-12-16T07:49:51.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:51 smithi007 bash[64479]: cluster 2023-12-16T07:49:51.273256+0000 mon.a (mon.0) 397 : cluster 1 Cluster is now healthy 2023-12-16T07:49:51.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:51 smithi007 bash[65864]: cluster 2023-12-16T07:49:51.273153+0000 mon.a (mon.0) 396 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50/630 objects degraded (7.937%), 15 pgs degraded) 2023-12-16T07:49:51.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:51 smithi007 bash[65864]: cluster 2023-12-16T07:49:51.273256+0000 mon.a (mon.0) 397 : cluster 1 Cluster is now healthy 2023-12-16T07:49:51.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:51 smithi169 bash[51638]: cluster 2023-12-16T07:49:51.273153+0000 mon.a (mon.0) 396 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50/630 objects degraded (7.937%), 15 pgs degraded) 2023-12-16T07:49:51.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:51 smithi169 bash[51638]: cluster 2023-12-16T07:49:51.273256+0000 mon.a (mon.0) 397 : cluster 1 Cluster is now healthy 2023-12-16T07:49:52.591 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:52 smithi007 bash[65864]: cluster 2023-12-16T07:49:51.221187+0000 mgr.y (mgr.25477) 246 : cluster 0 pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:49:52.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:52 smithi007 bash[64479]: cluster 2023-12-16T07:49:51.221187+0000 mgr.y (mgr.25477) 246 : cluster 0 pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:49:52.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:52 smithi169 bash[51638]: cluster 2023-12-16T07:49:51.221187+0000 mgr.y (mgr.25477) 246 : cluster 0 pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:49:54.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:54 smithi007 bash[64479]: audit 2023-12-16T07:49:53.211197+0000 mon.a (mon.0) 398 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:54.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:54 smithi007 bash[64479]: audit 2023-12-16T07:49:53.213325+0000 mon.a (mon.0) 399 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:49:54.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:54 smithi007 bash[64479]: cluster 2023-12-16T07:49:53.221900+0000 mgr.y (mgr.25477) 247 : cluster 0 pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:49:54.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:54 smithi007 bash[65864]: audit 2023-12-16T07:49:53.211197+0000 mon.a (mon.0) 398 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:54.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:54 smithi007 bash[65864]: audit 2023-12-16T07:49:53.213325+0000 mon.a (mon.0) 399 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:49:54.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:54 smithi007 bash[65864]: cluster 2023-12-16T07:49:53.221900+0000 mgr.y (mgr.25477) 247 : cluster 0 pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:49:54.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:54 smithi169 bash[51638]: audit 2023-12-16T07:49:53.211197+0000 mon.a (mon.0) 398 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:54.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:54 smithi169 bash[51638]: audit 2023-12-16T07:49:53.213325+0000 mon.a (mon.0) 399 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:49:54.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:54 smithi169 bash[51638]: cluster 2023-12-16T07:49:53.221900+0000 mgr.y (mgr.25477) 247 : cluster 0 pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:49:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[65864]: audit 2023-12-16T07:49:54.291231+0000 mgr.y (mgr.25477) 248 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[65864]: audit 2023-12-16T07:49:54.687412+0000 mon.a (mon.0) 400 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[65864]: audit 2023-12-16T07:49:54.698530+0000 mon.a (mon.0) 401 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:55.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[65864]: audit 2023-12-16T07:49:54.700630+0000 mon.a (mon.0) 402 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:49:55.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[65864]: audit 2023-12-16T07:49:54.702090+0000 mon.a (mon.0) 403 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:49:55.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[65864]: audit 2023-12-16T07:49:54.709813+0000 mon.a (mon.0) 404 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:55.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[65864]: audit 2023-12-16T07:49:54.796358+0000 mon.a (mon.0) 405 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:49:55.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[65864]: audit 2023-12-16T07:49:54.801962+0000 mon.a (mon.0) 406 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:55.573 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[65864]: audit 2023-12-16T07:49:54.805687+0000 mon.a (mon.0) 407 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:55.574 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[65864]: audit 2023-12-16T07:49:54.808459+0000 mon.a (mon.0) 408 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:55.574 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[65864]: audit 2023-12-16T07:49:54.811990+0000 mon.a (mon.0) 409 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-12-16T07:49:55.574 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[64479]: audit 2023-12-16T07:49:54.291231+0000 mgr.y (mgr.25477) 248 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:55.574 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[64479]: audit 2023-12-16T07:49:54.687412+0000 mon.a (mon.0) 400 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:55.575 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[64479]: audit 2023-12-16T07:49:54.698530+0000 mon.a (mon.0) 401 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:55.575 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[64479]: audit 2023-12-16T07:49:54.700630+0000 mon.a (mon.0) 402 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:49:55.575 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[64479]: audit 2023-12-16T07:49:54.702090+0000 mon.a (mon.0) 403 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:49:55.575 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[64479]: audit 2023-12-16T07:49:54.709813+0000 mon.a (mon.0) 404 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:55.576 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[64479]: audit 2023-12-16T07:49:54.796358+0000 mon.a (mon.0) 405 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:49:55.576 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[64479]: audit 2023-12-16T07:49:54.801962+0000 mon.a (mon.0) 406 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:55.576 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[64479]: audit 2023-12-16T07:49:54.805687+0000 mon.a (mon.0) 407 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:55.576 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[64479]: audit 2023-12-16T07:49:54.808459+0000 mon.a (mon.0) 408 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:55.576 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:55 smithi007 bash[64479]: audit 2023-12-16T07:49:54.811990+0000 mon.a (mon.0) 409 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-12-16T07:49:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:55 smithi169 bash[51638]: audit 2023-12-16T07:49:54.291231+0000 mgr.y (mgr.25477) 248 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:49:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:55 smithi169 bash[51638]: audit 2023-12-16T07:49:54.687412+0000 mon.a (mon.0) 400 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:55 smithi169 bash[51638]: audit 2023-12-16T07:49:54.698530+0000 mon.a (mon.0) 401 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:55 smithi169 bash[51638]: audit 2023-12-16T07:49:54.700630+0000 mon.a (mon.0) 402 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:49:55.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:55 smithi169 bash[51638]: audit 2023-12-16T07:49:54.702090+0000 mon.a (mon.0) 403 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:49:55.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:55 smithi169 bash[51638]: audit 2023-12-16T07:49:54.709813+0000 mon.a (mon.0) 404 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:55.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:55 smithi169 bash[51638]: audit 2023-12-16T07:49:54.796358+0000 mon.a (mon.0) 405 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:49:55.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:55 smithi169 bash[51638]: audit 2023-12-16T07:49:54.801962+0000 mon.a (mon.0) 406 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:55.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:55 smithi169 bash[51638]: audit 2023-12-16T07:49:54.805687+0000 mon.a (mon.0) 407 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:55.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:55 smithi169 bash[51638]: audit 2023-12-16T07:49:54.808459+0000 mon.a (mon.0) 408 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:49:55.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:55 smithi169 bash[51638]: audit 2023-12-16T07:49:54.811990+0000 mon.a (mon.0) 409 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-12-16T07:49:56.521 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:56 smithi169 bash[51638]: audit 2023-12-16T07:49:54.812370+0000 mgr.y (mgr.25477) 249 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-12-16T07:49:56.522 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:56 smithi169 bash[51638]: cephadm 2023-12-16T07:49:54.814376+0000 mgr.y (mgr.25477) 250 : cephadm 1 Upgrade: osd.4 is safe to restart 2023-12-16T07:49:56.522 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:56 smithi169 bash[51638]: cluster 2023-12-16T07:49:55.223603+0000 mgr.y (mgr.25477) 251 : cluster 0 pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:49:56.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:56 smithi007 bash[64479]: audit 2023-12-16T07:49:54.812370+0000 mgr.y (mgr.25477) 249 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-12-16T07:49:56.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:56 smithi007 bash[64479]: cephadm 2023-12-16T07:49:54.814376+0000 mgr.y (mgr.25477) 250 : cephadm 1 Upgrade: osd.4 is safe to restart 2023-12-16T07:49:56.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:56 smithi007 bash[64479]: cluster 2023-12-16T07:49:55.223603+0000 mgr.y (mgr.25477) 251 : cluster 0 pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:49:56.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:56 smithi007 bash[65864]: audit 2023-12-16T07:49:54.812370+0000 mgr.y (mgr.25477) 249 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2023-12-16T07:49:56.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:56 smithi007 bash[65864]: cephadm 2023-12-16T07:49:54.814376+0000 mgr.y (mgr.25477) 250 : cephadm 1 Upgrade: osd.4 is safe to restart 2023-12-16T07:49:56.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:56 smithi007 bash[65864]: cluster 2023-12-16T07:49:55.223603+0000 mgr.y (mgr.25477) 251 : cluster 0 pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:49:58.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:58 smithi007 bash[64479]: cluster 2023-12-16T07:49:57.224528+0000 mgr.y (mgr.25477) 252 : cluster 0 pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:49:58.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:58 smithi007 bash[64479]: cephadm 2023-12-16T07:49:57.235348+0000 mgr.y (mgr.25477) 253 : cephadm 1 Upgrade: Updating osd.4 2023-12-16T07:49:58.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:58 smithi007 bash[64479]: audit 2023-12-16T07:49:57.242359+0000 mon.a (mon.0) 410 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:58.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:58 smithi007 bash[64479]: audit 2023-12-16T07:49:57.244918+0000 mon.a (mon.0) 411 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-16T07:49:58.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:58 smithi007 bash[64479]: audit 2023-12-16T07:49:57.245966+0000 mon.a (mon.0) 412 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:49:58.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:58 smithi007 bash[64479]: cephadm 2023-12-16T07:49:57.249079+0000 mgr.y (mgr.25477) 254 : cephadm 1 Deploying daemon osd.4 on smithi169 2023-12-16T07:49:58.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:58 smithi007 bash[65864]: cluster 2023-12-16T07:49:57.224528+0000 mgr.y (mgr.25477) 252 : cluster 0 pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:49:58.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:58 smithi007 bash[65864]: cephadm 2023-12-16T07:49:57.235348+0000 mgr.y (mgr.25477) 253 : cephadm 1 Upgrade: Updating osd.4 2023-12-16T07:49:58.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:58 smithi007 bash[65864]: audit 2023-12-16T07:49:57.242359+0000 mon.a (mon.0) 410 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:58.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:58 smithi007 bash[65864]: audit 2023-12-16T07:49:57.244918+0000 mon.a (mon.0) 411 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-16T07:49:58.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:58 smithi007 bash[65864]: audit 2023-12-16T07:49:57.245966+0000 mon.a (mon.0) 412 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:49:58.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:58 smithi007 bash[65864]: cephadm 2023-12-16T07:49:57.249079+0000 mgr.y (mgr.25477) 254 : cephadm 1 Deploying daemon osd.4 on smithi169 2023-12-16T07:49:58.570 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:58 smithi169 bash[51638]: cluster 2023-12-16T07:49:57.224528+0000 mgr.y (mgr.25477) 252 : cluster 0 pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:49:58.570 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:58 smithi169 bash[51638]: cephadm 2023-12-16T07:49:57.235348+0000 mgr.y (mgr.25477) 253 : cephadm 1 Upgrade: Updating osd.4 2023-12-16T07:49:58.570 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:58 smithi169 bash[51638]: audit 2023-12-16T07:49:57.242359+0000 mon.a (mon.0) 410 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:58.570 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:58 smithi169 bash[51638]: audit 2023-12-16T07:49:57.244918+0000 mon.a (mon.0) 411 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-12-16T07:49:58.570 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:58 smithi169 bash[51638]: audit 2023-12-16T07:49:57.245966+0000 mon.a (mon.0) 412 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:49:58.570 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:58 smithi169 bash[51638]: cephadm 2023-12-16T07:49:57.249079+0000 mgr.y (mgr.25477) 254 : cephadm 1 Deploying daemon osd.4 on smithi169 2023-12-16T07:49:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:49:59 smithi169 bash[51638]: audit 2023-12-16T07:49:58.420447+0000 mon.a (mon.0) 413 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:59.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:49:59 smithi007 bash[64479]: audit 2023-12-16T07:49:58.420447+0000 mon.a (mon.0) 413 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:49:59.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:49:59 smithi007 bash[65864]: audit 2023-12-16T07:49:58.420447+0000 mon.a (mon.0) 413 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:00.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:50:00 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:50:00] "GET /metrics HTTP/1.1" 200 38408 "" "Prometheus/2.33.4" 2023-12-16T07:50:00.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:00 smithi007 bash[64479]: cluster 2023-12-16T07:49:59.225850+0000 mgr.y (mgr.25477) 255 : cluster 0 pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:50:00.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:00 smithi007 bash[64479]: cluster 2023-12-16T07:50:00.000130+0000 mon.a (mon.0) 414 : cluster 1 overall HEALTH_OK 2023-12-16T07:50:00.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:00 smithi007 bash[65864]: cluster 2023-12-16T07:49:59.225850+0000 mgr.y (mgr.25477) 255 : cluster 0 pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:50:00.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:00 smithi007 bash[65864]: cluster 2023-12-16T07:50:00.000130+0000 mon.a (mon.0) 414 : cluster 1 overall HEALTH_OK 2023-12-16T07:50:00.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:00 smithi169 bash[51638]: cluster 2023-12-16T07:49:59.225850+0000 mgr.y (mgr.25477) 255 : cluster 0 pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2023-12-16T07:50:00.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:00 smithi169 bash[51638]: cluster 2023-12-16T07:50:00.000130+0000 mon.a (mon.0) 414 : cluster 1 overall HEALTH_OK 2023-12-16T07:50:01.433 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:01 smithi169 systemd[1]: Stopping Ceph osd.4 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:50:01.933 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:01 smithi169 bash[28185]: debug 2023-12-16T07:50:01.436+0000 7f90eed91700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:50:01.933 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:01 smithi169 bash[28185]: debug 2023-12-16T07:50:01.436+0000 7f90eed91700 -1 osd.4 123 *** Got signal Terminated *** 2023-12-16T07:50:01.934 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:01 smithi169 bash[28185]: debug 2023-12-16T07:50:01.436+0000 7f90eed91700 -1 osd.4 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:50:02.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:02 smithi169 bash[51638]: cluster 2023-12-16T07:50:01.227131+0000 mgr.y (mgr.25477) 256 : cluster 0 pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-12-16T07:50:02.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:02 smithi169 bash[51638]: cluster 2023-12-16T07:50:01.438409+0000 mon.a (mon.0) 415 : cluster 1 osd.4 marked itself down and dead 2023-12-16T07:50:02.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:02 smithi007 bash[64479]: cluster 2023-12-16T07:50:01.227131+0000 mgr.y (mgr.25477) 256 : cluster 0 pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-12-16T07:50:02.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:02 smithi007 bash[64479]: cluster 2023-12-16T07:50:01.438409+0000 mon.a (mon.0) 415 : cluster 1 osd.4 marked itself down and dead 2023-12-16T07:50:02.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:02 smithi007 bash[65864]: cluster 2023-12-16T07:50:01.227131+0000 mgr.y (mgr.25477) 256 : cluster 0 pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-12-16T07:50:02.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:02 smithi007 bash[65864]: cluster 2023-12-16T07:50:01.438409+0000 mon.a (mon.0) 415 : cluster 1 osd.4 marked itself down and dead 2023-12-16T07:50:03.681 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:03 smithi169 bash[53480]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-4 2023-12-16T07:50:03.681 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:03 smithi169 bash[53550]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-4 2023-12-16T07:50:03.681 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:03 smithi169 bash[51638]: cluster 2023-12-16T07:50:02.424164+0000 mon.a (mon.0) 416 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:50:03.681 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:03 smithi169 bash[51638]: cluster 2023-12-16T07:50:02.437086+0000 osd.6 (osd.6) 28 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:03.681 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:03 smithi169 bash[51638]: cluster 2023-12-16T07:50:02.440321+0000 mon.a (mon.0) 417 : cluster 0 osdmap e124: 8 total, 7 up, 8 in 2023-12-16T07:50:03.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:03 smithi169 bash[51638]: cluster 2023-12-16T07:50:02.441235+0000 osd.5 (osd.5) 30 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:03.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:03 smithi169 bash[51638]: cluster 2023-12-16T07:50:02.442595+0000 osd.7 (osd.7) 28 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:03.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:03 smithi169 bash[51638]: cluster 2023-12-16T07:50:02.443578+0000 osd.6 (osd.6) 29 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:03.682 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:03 smithi169 bash[51638]: cluster 2023-12-16T07:50:02.450486+0000 osd.7 (osd.7) 29 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:03.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:03 smithi007 bash[65864]: cluster 2023-12-16T07:50:02.424164+0000 mon.a (mon.0) 416 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:50:03.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:03 smithi007 bash[65864]: cluster 2023-12-16T07:50:02.437086+0000 osd.6 (osd.6) 28 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:03.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:03 smithi007 bash[65864]: cluster 2023-12-16T07:50:02.440321+0000 mon.a (mon.0) 417 : cluster 0 osdmap e124: 8 total, 7 up, 8 in 2023-12-16T07:50:03.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:03 smithi007 bash[65864]: cluster 2023-12-16T07:50:02.441235+0000 osd.5 (osd.5) 30 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:03.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:03 smithi007 bash[65864]: cluster 2023-12-16T07:50:02.442595+0000 osd.7 (osd.7) 28 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:03.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:03 smithi007 bash[65864]: cluster 2023-12-16T07:50:02.443578+0000 osd.6 (osd.6) 29 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:03.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:03 smithi007 bash[65864]: cluster 2023-12-16T07:50:02.450486+0000 osd.7 (osd.7) 29 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:03.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:03 smithi007 bash[64479]: cluster 2023-12-16T07:50:02.424164+0000 mon.a (mon.0) 416 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:50:03.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:03 smithi007 bash[64479]: cluster 2023-12-16T07:50:02.437086+0000 osd.6 (osd.6) 28 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:03.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:03 smithi007 bash[64479]: cluster 2023-12-16T07:50:02.440321+0000 mon.a (mon.0) 417 : cluster 0 osdmap e124: 8 total, 7 up, 8 in 2023-12-16T07:50:03.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:03 smithi007 bash[64479]: cluster 2023-12-16T07:50:02.441235+0000 osd.5 (osd.5) 30 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:03.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:03 smithi007 bash[64479]: cluster 2023-12-16T07:50:02.442595+0000 osd.7 (osd.7) 28 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:03.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:03 smithi007 bash[64479]: cluster 2023-12-16T07:50:02.443578+0000 osd.6 (osd.6) 29 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:03.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:03 smithi007 bash[64479]: cluster 2023-12-16T07:50:02.450486+0000 osd.7 (osd.7) 29 : cluster 3 failed to encode map e124 with expected crc 2023-12-16T07:50:04.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[65864]: cluster 2023-12-16T07:50:03.227881+0000 mgr.y (mgr.25477) 257 : cluster 0 pgmap v149: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:50:04.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[65864]: cluster 2023-12-16T07:50:03.447029+0000 mon.a (mon.0) 418 : cluster 0 osdmap e125: 8 total, 7 up, 8 in 2023-12-16T07:50:04.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[65864]: cluster 2023-12-16T07:50:03.448541+0000 osd.6 (osd.6) 30 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[65864]: cluster 2023-12-16T07:50:03.451764+0000 osd.6 (osd.6) 31 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[65864]: cluster 2023-12-16T07:50:03.452980+0000 osd.7 (osd.7) 30 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[65864]: cluster 2023-12-16T07:50:03.454382+0000 osd.5 (osd.5) 31 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[65864]: cluster 2023-12-16T07:50:03.455715+0000 osd.7 (osd.7) 31 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[65864]: cluster 2023-12-16T07:50:03.455862+0000 osd.6 (osd.6) 32 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[65864]: cluster 2023-12-16T07:50:03.456981+0000 osd.7 (osd.7) 32 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[65864]: cluster 2023-12-16T07:50:03.457241+0000 osd.5 (osd.5) 32 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[65864]: cluster 2023-12-16T07:50:03.459724+0000 osd.5 (osd.5) 33 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[64479]: cluster 2023-12-16T07:50:03.227881+0000 mgr.y (mgr.25477) 257 : cluster 0 pgmap v149: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:50:04.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[64479]: cluster 2023-12-16T07:50:03.447029+0000 mon.a (mon.0) 418 : cluster 0 osdmap e125: 8 total, 7 up, 8 in 2023-12-16T07:50:04.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[64479]: cluster 2023-12-16T07:50:03.448541+0000 osd.6 (osd.6) 30 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[64479]: cluster 2023-12-16T07:50:03.451764+0000 osd.6 (osd.6) 31 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[64479]: cluster 2023-12-16T07:50:03.452980+0000 osd.7 (osd.7) 30 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[64479]: cluster 2023-12-16T07:50:03.454382+0000 osd.5 (osd.5) 31 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[64479]: cluster 2023-12-16T07:50:03.455715+0000 osd.7 (osd.7) 31 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[64479]: cluster 2023-12-16T07:50:03.455862+0000 osd.6 (osd.6) 32 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[64479]: cluster 2023-12-16T07:50:03.456981+0000 osd.7 (osd.7) 32 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[64479]: cluster 2023-12-16T07:50:03.457241+0000 osd.5 (osd.5) 32 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.571 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:04 smithi007 bash[64479]: cluster 2023-12-16T07:50:03.459724+0000 osd.5 (osd.5) 33 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:04 smithi169 bash[51638]: cluster 2023-12-16T07:50:03.227881+0000 mgr.y (mgr.25477) 257 : cluster 0 pgmap v149: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:50:04.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:04 smithi169 bash[51638]: cluster 2023-12-16T07:50:03.447029+0000 mon.a (mon.0) 418 : cluster 0 osdmap e125: 8 total, 7 up, 8 in 2023-12-16T07:50:04.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:04 smithi169 bash[51638]: cluster 2023-12-16T07:50:03.448541+0000 osd.6 (osd.6) 30 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:04 smithi169 bash[51638]: cluster 2023-12-16T07:50:03.451764+0000 osd.6 (osd.6) 31 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:04 smithi169 bash[51638]: cluster 2023-12-16T07:50:03.452980+0000 osd.7 (osd.7) 30 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:04 smithi169 bash[51638]: cluster 2023-12-16T07:50:03.454382+0000 osd.5 (osd.5) 31 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:04 smithi169 bash[51638]: cluster 2023-12-16T07:50:03.455715+0000 osd.7 (osd.7) 31 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:04 smithi169 bash[51638]: cluster 2023-12-16T07:50:03.455862+0000 osd.6 (osd.6) 32 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:04 smithi169 bash[51638]: cluster 2023-12-16T07:50:03.456981+0000 osd.7 (osd.7) 32 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:04 smithi169 bash[51638]: cluster 2023-12-16T07:50:03.457241+0000 osd.5 (osd.5) 32 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:04.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:04 smithi169 bash[51638]: cluster 2023-12-16T07:50:03.459724+0000 osd.5 (osd.5) 33 : cluster 3 failed to encode map e125 with expected crc 2023-12-16T07:50:05.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:05 smithi007 bash[64479]: audit 2023-12-16T07:50:04.299437+0000 mgr.y (mgr.25477) 258 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:05.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:05 smithi007 bash[65864]: audit 2023-12-16T07:50:04.299437+0000 mgr.y (mgr.25477) 258 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:05.820 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:05 smithi169 bash[51638]: audit 2023-12-16T07:50:04.299437+0000 mgr.y (mgr.25477) 258 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:05.820 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:05 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.4.service: Succeeded. 2023-12-16T07:50:05.820 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:05 smithi169 systemd[1]: Stopped Ceph osd.4 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:50:06.183 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:05 smithi169 systemd[1]: Started Ceph osd.4 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:50:06.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[64479]: cluster 2023-12-16T07:50:05.229749+0000 mgr.y (mgr.25477) 259 : cluster 0 pgmap v151: 161 pgs: 22 active+undersized+degraded, 29 active+undersized, 6 peering, 3 stale+active+clean, 101 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2023-12-16T07:50:06.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[64479]: cluster 2023-12-16T07:50:05.444961+0000 mon.a (mon.0) 419 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-12-16T07:50:06.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[64479]: cluster 2023-12-16T07:50:05.445033+0000 mon.a (mon.0) 420 : cluster 3 Health check failed: Degraded data redundancy: 89/630 objects degraded (14.127%), 22 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:06.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[64479]: audit 2023-12-16T07:50:05.869021+0000 mon.a (mon.0) 421 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[64479]: audit 2023-12-16T07:50:05.880093+0000 mon.a (mon.0) 422 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[64479]: audit 2023-12-16T07:50:06.410918+0000 mon.a (mon.0) 423 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[64479]: audit 2023-12-16T07:50:06.416737+0000 mon.a (mon.0) 424 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[64479]: audit 2023-12-16T07:50:06.425204+0000 mon.a (mon.0) 425 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[64479]: audit 2023-12-16T07:50:06.431799+0000 mon.a (mon.0) 426 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[65864]: cluster 2023-12-16T07:50:05.229749+0000 mgr.y (mgr.25477) 259 : cluster 0 pgmap v151: 161 pgs: 22 active+undersized+degraded, 29 active+undersized, 6 peering, 3 stale+active+clean, 101 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2023-12-16T07:50:06.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[65864]: cluster 2023-12-16T07:50:05.444961+0000 mon.a (mon.0) 419 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-12-16T07:50:06.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[65864]: cluster 2023-12-16T07:50:05.445033+0000 mon.a (mon.0) 420 : cluster 3 Health check failed: Degraded data redundancy: 89/630 objects degraded (14.127%), 22 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:06.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[65864]: audit 2023-12-16T07:50:05.869021+0000 mon.a (mon.0) 421 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[65864]: audit 2023-12-16T07:50:05.880093+0000 mon.a (mon.0) 422 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[65864]: audit 2023-12-16T07:50:06.410918+0000 mon.a (mon.0) 423 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[65864]: audit 2023-12-16T07:50:06.416737+0000 mon.a (mon.0) 424 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[65864]: audit 2023-12-16T07:50:06.425204+0000 mon.a (mon.0) 425 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:06 smithi007 bash[65864]: audit 2023-12-16T07:50:06.431799+0000 mon.a (mon.0) 426 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:06 smithi169 bash[51638]: cluster 2023-12-16T07:50:05.229749+0000 mgr.y (mgr.25477) 259 : cluster 0 pgmap v151: 161 pgs: 22 active+undersized+degraded, 29 active+undersized, 6 peering, 3 stale+active+clean, 101 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 89/630 objects degraded (14.127%) 2023-12-16T07:50:06.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:06 smithi169 bash[51638]: cluster 2023-12-16T07:50:05.444961+0000 mon.a (mon.0) 419 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-12-16T07:50:06.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:06 smithi169 bash[51638]: cluster 2023-12-16T07:50:05.445033+0000 mon.a (mon.0) 420 : cluster 3 Health check failed: Degraded data redundancy: 89/630 objects degraded (14.127%), 22 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:06.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:06 smithi169 bash[51638]: audit 2023-12-16T07:50:05.869021+0000 mon.a (mon.0) 421 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:06 smithi169 bash[51638]: audit 2023-12-16T07:50:05.880093+0000 mon.a (mon.0) 422 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:06 smithi169 bash[51638]: audit 2023-12-16T07:50:06.410918+0000 mon.a (mon.0) 423 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:06 smithi169 bash[51638]: audit 2023-12-16T07:50:06.416737+0000 mon.a (mon.0) 424 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:06 smithi169 bash[51638]: audit 2023-12-16T07:50:06.425204+0000 mon.a (mon.0) 425 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:06.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:06 smithi169 bash[51638]: audit 2023-12-16T07:50:06.431799+0000 mon.a (mon.0) 426 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:07.683 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:07 smithi169 bash[53717]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-16T07:50:07.683 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:07 smithi169 bash[53717]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2023-12-16T07:50:07.683 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:07 smithi169 bash[53717]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2023-12-16T07:50:07.683 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:07 smithi169 bash[53717]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-12-16T07:50:07.683 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:07 smithi169 bash[53717]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2023-12-16T07:50:07.683 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:07 smithi169 bash[53717]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-16T07:50:07.684 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:07 smithi169 bash[53717]: --> ceph-volume raw activate successful for osd ID: 4 2023-12-16T07:50:08.754 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:08 smithi169 bash[51638]: cluster 2023-12-16T07:50:07.231097+0000 mgr.y (mgr.25477) 260 : cluster 0 pgmap v152: 161 pgs: 27 active+undersized+degraded, 35 active+undersized, 6 peering, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 103/630 objects degraded (16.349%) 2023-12-16T07:50:08.754 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:08 smithi169 bash[51638]: audit 2023-12-16T07:50:08.211467+0000 mon.a (mon.0) 427 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:08.754 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:08 smithi169 bash[51638]: audit 2023-12-16T07:50:08.214174+0000 mon.a (mon.0) 428 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:50:08.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:08 smithi007 bash[64479]: cluster 2023-12-16T07:50:07.231097+0000 mgr.y (mgr.25477) 260 : cluster 0 pgmap v152: 161 pgs: 27 active+undersized+degraded, 35 active+undersized, 6 peering, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 103/630 objects degraded (16.349%) 2023-12-16T07:50:08.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:08 smithi007 bash[64479]: audit 2023-12-16T07:50:08.211467+0000 mon.a (mon.0) 427 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:08.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:08 smithi007 bash[64479]: audit 2023-12-16T07:50:08.214174+0000 mon.a (mon.0) 428 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:50:08.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:08 smithi007 bash[65864]: cluster 2023-12-16T07:50:07.231097+0000 mgr.y (mgr.25477) 260 : cluster 0 pgmap v152: 161 pgs: 27 active+undersized+degraded, 35 active+undersized, 6 peering, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 103/630 objects degraded (16.349%) 2023-12-16T07:50:08.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:08 smithi007 bash[65864]: audit 2023-12-16T07:50:08.211467+0000 mon.a (mon.0) 427 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:08.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:08 smithi007 bash[65864]: audit 2023-12-16T07:50:08.214174+0000 mon.a (mon.0) 428 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:50:09.183 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:08 smithi169 bash[54022]: debug 2023-12-16T07:50:08.796+0000 7fc6498dc700 1 -- 172.21.15.169:0/923828665 <== mon.1 v2:172.21.15.7:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560fc30236c0 con 0x560fc30a8c00 2023-12-16T07:50:09.683 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:09 smithi169 bash[54022]: debug 2023-12-16T07:50:09.336+0000 7fc652a62640 -1 Falling back to public interface 2023-12-16T07:50:10.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:09 smithi169 bash[51638]: cluster 2023-12-16T07:50:09.232669+0000 mgr.y (mgr.25477) 261 : cluster 0 pgmap v153: 161 pgs: 29 active+undersized+degraded, 39 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 109/630 objects degraded (17.302%) 2023-12-16T07:50:10.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:10 smithi007 bash[65864]: cluster 2023-12-16T07:50:09.232669+0000 mgr.y (mgr.25477) 261 : cluster 0 pgmap v153: 161 pgs: 29 active+undersized+degraded, 39 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 109/630 objects degraded (17.302%) 2023-12-16T07:50:10.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:09 smithi007 bash[64479]: cluster 2023-12-16T07:50:09.232669+0000 mgr.y (mgr.25477) 261 : cluster 0 pgmap v153: 161 pgs: 29 active+undersized+degraded, 39 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 109/630 objects degraded (17.302%) 2023-12-16T07:50:10.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:50:10 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:50:10] "GET /metrics HTTP/1.1" 200 38410 "" "Prometheus/2.33.4" 2023-12-16T07:50:11.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:10 smithi169 bash[51638]: cluster 2023-12-16T07:50:09.857108+0000 mon.a (mon.0) 429 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-12-16T07:50:11.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:10 smithi007 bash[64479]: cluster 2023-12-16T07:50:09.857108+0000 mon.a (mon.0) 429 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-12-16T07:50:11.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:10 smithi007 bash[65864]: cluster 2023-12-16T07:50:09.857108+0000 mon.a (mon.0) 429 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-12-16T07:50:12.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:11 smithi169 bash[51638]: cluster 2023-12-16T07:50:11.234314+0000 mgr.y (mgr.25477) 262 : cluster 0 pgmap v154: 161 pgs: 29 active+undersized+degraded, 39 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 109/630 objects degraded (17.302%) 2023-12-16T07:50:12.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:11 smithi007 bash[64479]: cluster 2023-12-16T07:50:11.234314+0000 mgr.y (mgr.25477) 262 : cluster 0 pgmap v154: 161 pgs: 29 active+undersized+degraded, 39 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 109/630 objects degraded (17.302%) 2023-12-16T07:50:12.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:11 smithi007 bash[65864]: cluster 2023-12-16T07:50:11.234314+0000 mgr.y (mgr.25477) 262 : cluster 0 pgmap v154: 161 pgs: 29 active+undersized+degraded, 39 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 109/630 objects degraded (17.302%) 2023-12-16T07:50:12.683 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:12 smithi169 bash[54022]: debug 2023-12-16T07:50:12.307+0000 7fc652a62640 -1 osd.4 0 read_superblock omap replica is missing. 2023-12-16T07:50:12.683 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:12 smithi169 bash[54022]: debug 2023-12-16T07:50:12.359+0000 7fc652a62640 -1 osd.4 123 log_to_monitors true 2023-12-16T07:50:13.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:12 smithi169 bash[51638]: audit 2023-12-16T07:50:12.366369+0000 mon.a (mon.0) 430 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T07:50:13.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:12 smithi169 bash[51638]: audit 2023-12-16T07:50:12.367782+0000 mon.b (mon.2) 3 : audit 1 from='osd.4 [v2:172.21.15.169:6800/1152555513,v1:172.21.15.169:6801/1152555513]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T07:50:13.183 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:50:12 smithi169 bash[54022]: debug 2023-12-16T07:50:12.927+0000 7fc6426d0700 -1 osd.4 123 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:50:13.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:12 smithi007 bash[64479]: audit 2023-12-16T07:50:12.366369+0000 mon.a (mon.0) 430 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T07:50:13.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:12 smithi007 bash[64479]: audit 2023-12-16T07:50:12.367782+0000 mon.b (mon.2) 3 : audit 1 from='osd.4 [v2:172.21.15.169:6800/1152555513,v1:172.21.15.169:6801/1152555513]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T07:50:13.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:12 smithi007 bash[65864]: audit 2023-12-16T07:50:12.366369+0000 mon.a (mon.0) 430 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T07:50:13.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:12 smithi007 bash[65864]: audit 2023-12-16T07:50:12.367782+0000 mon.b (mon.2) 3 : audit 1 from='osd.4 [v2:172.21.15.169:6800/1152555513,v1:172.21.15.169:6801/1152555513]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-12-16T07:50:14.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:13 smithi169 bash[51638]: audit 2023-12-16T07:50:12.882658+0000 mon.a (mon.0) 431 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-16T07:50:14.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:13 smithi169 bash[51638]: cluster 2023-12-16T07:50:12.886075+0000 osd.6 (osd.6) 33 : cluster 3 failed to encode map e126 with expected crc 2023-12-16T07:50:14.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:13 smithi169 bash[51638]: cluster 2023-12-16T07:50:12.886668+0000 osd.7 (osd.7) 33 : cluster 3 failed to encode map e126 with expected crc 2023-12-16T07:50:14.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:13 smithi169 bash[51638]: cluster 2023-12-16T07:50:12.887150+0000 mon.a (mon.0) 432 : cluster 0 osdmap e126: 8 total, 7 up, 8 in 2023-12-16T07:50:14.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:13 smithi169 bash[51638]: audit 2023-12-16T07:50:12.888865+0000 mon.a (mon.0) 433 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:50:14.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:13 smithi169 bash[51638]: audit 2023-12-16T07:50:12.890023+0000 mon.b (mon.2) 4 : audit 1 from='osd.4 [v2:172.21.15.169:6800/1152555513,v1:172.21.15.169:6801/1152555513]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:50:14.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:13 smithi169 bash[51638]: cluster 2023-12-16T07:50:13.235464+0000 mgr.y (mgr.25477) 263 : cluster 0 pgmap v156: 161 pgs: 29 active+undersized+degraded, 39 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 109/630 objects degraded (17.302%) 2023-12-16T07:50:14.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:13 smithi169 bash[51638]: cluster 2023-12-16T07:50:13.855122+0000 mon.a (mon.0) 434 : cluster 3 Health check update: Degraded data redundancy: 109/630 objects degraded (17.302%), 29 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:14.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:13 smithi169 bash[51638]: audit 2023-12-16T07:50:13.877684+0000 mon.a (mon.0) 435 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:14.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[65864]: audit 2023-12-16T07:50:12.882658+0000 mon.a (mon.0) 431 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-16T07:50:14.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[65864]: cluster 2023-12-16T07:50:12.886075+0000 osd.6 (osd.6) 33 : cluster 3 failed to encode map e126 with expected crc 2023-12-16T07:50:14.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[65864]: cluster 2023-12-16T07:50:12.886668+0000 osd.7 (osd.7) 33 : cluster 3 failed to encode map e126 with expected crc 2023-12-16T07:50:14.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[65864]: cluster 2023-12-16T07:50:12.887150+0000 mon.a (mon.0) 432 : cluster 0 osdmap e126: 8 total, 7 up, 8 in 2023-12-16T07:50:14.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[65864]: audit 2023-12-16T07:50:12.888865+0000 mon.a (mon.0) 433 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:50:14.314 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[65864]: audit 2023-12-16T07:50:12.890023+0000 mon.b (mon.2) 4 : audit 1 from='osd.4 [v2:172.21.15.169:6800/1152555513,v1:172.21.15.169:6801/1152555513]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:50:14.314 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[65864]: cluster 2023-12-16T07:50:13.235464+0000 mgr.y (mgr.25477) 263 : cluster 0 pgmap v156: 161 pgs: 29 active+undersized+degraded, 39 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 109/630 objects degraded (17.302%) 2023-12-16T07:50:14.314 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[65864]: cluster 2023-12-16T07:50:13.855122+0000 mon.a (mon.0) 434 : cluster 3 Health check update: Degraded data redundancy: 109/630 objects degraded (17.302%), 29 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:14.314 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[65864]: audit 2023-12-16T07:50:13.877684+0000 mon.a (mon.0) 435 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:14.314 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[64479]: audit 2023-12-16T07:50:12.882658+0000 mon.a (mon.0) 431 : audit 1 from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-12-16T07:50:14.314 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[64479]: cluster 2023-12-16T07:50:12.886075+0000 osd.6 (osd.6) 33 : cluster 3 failed to encode map e126 with expected crc 2023-12-16T07:50:14.314 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[64479]: cluster 2023-12-16T07:50:12.886668+0000 osd.7 (osd.7) 33 : cluster 3 failed to encode map e126 with expected crc 2023-12-16T07:50:14.314 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[64479]: cluster 2023-12-16T07:50:12.887150+0000 mon.a (mon.0) 432 : cluster 0 osdmap e126: 8 total, 7 up, 8 in 2023-12-16T07:50:14.314 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[64479]: audit 2023-12-16T07:50:12.888865+0000 mon.a (mon.0) 433 : audit 1 from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:50:14.314 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[64479]: audit 2023-12-16T07:50:12.890023+0000 mon.b (mon.2) 4 : audit 1 from='osd.4 [v2:172.21.15.169:6800/1152555513,v1:172.21.15.169:6801/1152555513]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:50:14.314 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[64479]: cluster 2023-12-16T07:50:13.235464+0000 mgr.y (mgr.25477) 263 : cluster 0 pgmap v156: 161 pgs: 29 active+undersized+degraded, 39 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 109/630 objects degraded (17.302%) 2023-12-16T07:50:14.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[64479]: cluster 2023-12-16T07:50:13.855122+0000 mon.a (mon.0) 434 : cluster 3 Health check update: Degraded data redundancy: 109/630 objects degraded (17.302%), 29 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:14.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:13 smithi007 bash[64479]: audit 2023-12-16T07:50:13.877684+0000 mon.a (mon.0) 435 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:15.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:14 smithi169 bash[51638]: cluster 2023-12-16T07:50:13.883505+0000 mon.a (mon.0) 436 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:50:15.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:14 smithi169 bash[51638]: audit 2023-12-16T07:50:13.886917+0000 mon.a (mon.0) 437 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:15.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:14 smithi169 bash[51638]: cluster 2023-12-16T07:50:13.894895+0000 osd.5 (osd.5) 34 : cluster 3 failed to encode map e127 with expected crc 2023-12-16T07:50:15.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:14 smithi169 bash[51638]: cluster 2023-12-16T07:50:13.897140+0000 mon.a (mon.0) 438 : cluster 1 osd.4 [v2:172.21.15.169:6800/1152555513,v1:172.21.15.169:6801/1152555513] boot 2023-12-16T07:50:15.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:14 smithi169 bash[51638]: cluster 2023-12-16T07:50:13.897231+0000 mon.a (mon.0) 439 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2023-12-16T07:50:15.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:14 smithi169 bash[51638]: audit 2023-12-16T07:50:13.898259+0000 mon.a (mon.0) 440 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:50:15.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:14 smithi169 bash[51638]: cluster 2023-12-16T07:50:13.901417+0000 osd.5 (osd.5) 35 : cluster 3 failed to encode map e127 with expected crc 2023-12-16T07:50:15.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:14 smithi169 bash[51638]: cluster 2023-12-16T07:50:13.904390+0000 osd.7 (osd.7) 34 : cluster 3 failed to encode map e127 with expected crc 2023-12-16T07:50:15.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:14 smithi169 bash[51638]: audit 2023-12-16T07:50:14.309350+0000 mgr.y (mgr.25477) 264 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:15.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[64479]: cluster 2023-12-16T07:50:13.883505+0000 mon.a (mon.0) 436 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:50:15.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[64479]: audit 2023-12-16T07:50:13.886917+0000 mon.a (mon.0) 437 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:15.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[64479]: cluster 2023-12-16T07:50:13.894895+0000 osd.5 (osd.5) 34 : cluster 3 failed to encode map e127 with expected crc 2023-12-16T07:50:15.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[64479]: cluster 2023-12-16T07:50:13.897140+0000 mon.a (mon.0) 438 : cluster 1 osd.4 [v2:172.21.15.169:6800/1152555513,v1:172.21.15.169:6801/1152555513] boot 2023-12-16T07:50:15.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[64479]: cluster 2023-12-16T07:50:13.897231+0000 mon.a (mon.0) 439 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2023-12-16T07:50:15.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[64479]: audit 2023-12-16T07:50:13.898259+0000 mon.a (mon.0) 440 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:50:15.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[64479]: cluster 2023-12-16T07:50:13.901417+0000 osd.5 (osd.5) 35 : cluster 3 failed to encode map e127 with expected crc 2023-12-16T07:50:15.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[64479]: cluster 2023-12-16T07:50:13.904390+0000 osd.7 (osd.7) 34 : cluster 3 failed to encode map e127 with expected crc 2023-12-16T07:50:15.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[64479]: audit 2023-12-16T07:50:14.309350+0000 mgr.y (mgr.25477) 264 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:15.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[65864]: cluster 2023-12-16T07:50:13.883505+0000 mon.a (mon.0) 436 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:50:15.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[65864]: audit 2023-12-16T07:50:13.886917+0000 mon.a (mon.0) 437 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:15.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[65864]: cluster 2023-12-16T07:50:13.894895+0000 osd.5 (osd.5) 34 : cluster 3 failed to encode map e127 with expected crc 2023-12-16T07:50:15.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[65864]: cluster 2023-12-16T07:50:13.897140+0000 mon.a (mon.0) 438 : cluster 1 osd.4 [v2:172.21.15.169:6800/1152555513,v1:172.21.15.169:6801/1152555513] boot 2023-12-16T07:50:15.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[65864]: cluster 2023-12-16T07:50:13.897231+0000 mon.a (mon.0) 439 : cluster 0 osdmap e127: 8 total, 8 up, 8 in 2023-12-16T07:50:15.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[65864]: audit 2023-12-16T07:50:13.898259+0000 mon.a (mon.0) 440 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-12-16T07:50:15.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[65864]: cluster 2023-12-16T07:50:13.901417+0000 osd.5 (osd.5) 35 : cluster 3 failed to encode map e127 with expected crc 2023-12-16T07:50:15.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[65864]: cluster 2023-12-16T07:50:13.904390+0000 osd.7 (osd.7) 34 : cluster 3 failed to encode map e127 with expected crc 2023-12-16T07:50:15.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:14 smithi007 bash[65864]: audit 2023-12-16T07:50:14.309350+0000 mgr.y (mgr.25477) 264 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:16.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:15 smithi169 bash[51638]: cluster 2023-12-16T07:50:14.897043+0000 osd.7 (osd.7) 35 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:15 smithi169 bash[51638]: cluster 2023-12-16T07:50:14.900004+0000 mon.a (mon.0) 441 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2023-12-16T07:50:16.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:15 smithi169 bash[51638]: cluster 2023-12-16T07:50:14.902846+0000 osd.6 (osd.6) 34 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:15 smithi169 bash[51638]: cluster 2023-12-16T07:50:14.903402+0000 osd.5 (osd.5) 36 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:15 smithi169 bash[51638]: cluster 2023-12-16T07:50:14.906088+0000 osd.6 (osd.6) 35 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:15 smithi169 bash[51638]: cluster 2023-12-16T07:50:14.908229+0000 osd.5 (osd.5) 37 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:15 smithi169 bash[51638]: cluster 2023-12-16T07:50:14.909506+0000 osd.6 (osd.6) 36 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:15 smithi169 bash[51638]: cluster 2023-12-16T07:50:14.910476+0000 osd.5 (osd.5) 38 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:15 smithi169 bash[51638]: cluster 2023-12-16T07:50:14.912197+0000 osd.5 (osd.5) 39 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:15 smithi169 bash[51638]: audit 2023-12-16T07:50:14.986596+0000 mon.a (mon.0) 442 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:16.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:15 smithi169 bash[51638]: audit 2023-12-16T07:50:14.999208+0000 mon.a (mon.0) 443 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:16.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:15 smithi169 bash[51638]: cluster 2023-12-16T07:50:15.236510+0000 mgr.y (mgr.25477) 265 : cluster 0 pgmap v159: 161 pgs: 34 peering, 13 active+undersized+degraded, 21 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 38/630 objects degraded (6.032%) 2023-12-16T07:50:16.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[64479]: cluster 2023-12-16T07:50:14.897043+0000 osd.7 (osd.7) 35 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[64479]: cluster 2023-12-16T07:50:14.900004+0000 mon.a (mon.0) 441 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2023-12-16T07:50:16.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[64479]: cluster 2023-12-16T07:50:14.902846+0000 osd.6 (osd.6) 34 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[64479]: cluster 2023-12-16T07:50:14.903402+0000 osd.5 (osd.5) 36 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[64479]: cluster 2023-12-16T07:50:14.906088+0000 osd.6 (osd.6) 35 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[64479]: cluster 2023-12-16T07:50:14.908229+0000 osd.5 (osd.5) 37 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[64479]: cluster 2023-12-16T07:50:14.909506+0000 osd.6 (osd.6) 36 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[64479]: cluster 2023-12-16T07:50:14.910476+0000 osd.5 (osd.5) 38 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[64479]: cluster 2023-12-16T07:50:14.912197+0000 osd.5 (osd.5) 39 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[64479]: audit 2023-12-16T07:50:14.986596+0000 mon.a (mon.0) 442 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:16.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[64479]: audit 2023-12-16T07:50:14.999208+0000 mon.a (mon.0) 443 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:16.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[64479]: cluster 2023-12-16T07:50:15.236510+0000 mgr.y (mgr.25477) 265 : cluster 0 pgmap v159: 161 pgs: 34 peering, 13 active+undersized+degraded, 21 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 38/630 objects degraded (6.032%) 2023-12-16T07:50:16.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[65864]: cluster 2023-12-16T07:50:14.897043+0000 osd.7 (osd.7) 35 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[65864]: cluster 2023-12-16T07:50:14.900004+0000 mon.a (mon.0) 441 : cluster 0 osdmap e128: 8 total, 8 up, 8 in 2023-12-16T07:50:16.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[65864]: cluster 2023-12-16T07:50:14.902846+0000 osd.6 (osd.6) 34 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[65864]: cluster 2023-12-16T07:50:14.903402+0000 osd.5 (osd.5) 36 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[65864]: cluster 2023-12-16T07:50:14.906088+0000 osd.6 (osd.6) 35 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[65864]: cluster 2023-12-16T07:50:14.908229+0000 osd.5 (osd.5) 37 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[65864]: cluster 2023-12-16T07:50:14.909506+0000 osd.6 (osd.6) 36 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[65864]: cluster 2023-12-16T07:50:14.910476+0000 osd.5 (osd.5) 38 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[65864]: cluster 2023-12-16T07:50:14.912197+0000 osd.5 (osd.5) 39 : cluster 3 failed to encode map e128 with expected crc 2023-12-16T07:50:16.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[65864]: audit 2023-12-16T07:50:14.986596+0000 mon.a (mon.0) 442 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:16.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[65864]: audit 2023-12-16T07:50:14.999208+0000 mon.a (mon.0) 443 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:16.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:15 smithi007 bash[65864]: cluster 2023-12-16T07:50:15.236510+0000 mgr.y (mgr.25477) 265 : cluster 0 pgmap v159: 161 pgs: 34 peering, 13 active+undersized+degraded, 21 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 38/630 objects degraded (6.032%) 2023-12-16T07:50:18.538 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:50:18.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:18 smithi007 bash[64479]: cluster 2023-12-16T07:50:17.237595+0000 mgr.y (mgr.25477) 266 : cluster 0 pgmap v160: 161 pgs: 34 peering, 10 active+undersized+degraded, 16 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2023-12-16T07:50:18.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:18 smithi007 bash[65864]: cluster 2023-12-16T07:50:17.237595+0000 mgr.y (mgr.25477) 266 : cluster 0 pgmap v160: 161 pgs: 34 peering, 10 active+undersized+degraded, 16 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2023-12-16T07:50:18.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:18 smithi169 bash[51638]: cluster 2023-12-16T07:50:17.237595+0000 mgr.y (mgr.25477) 266 : cluster 0 pgmap v160: 161 pgs: 34 peering, 10 active+undersized+degraded, 16 active+undersized, 101 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 29/630 objects degraded (4.603%) 2023-12-16T07:50:19.345 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:50:19.345 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (6m) 35s ago 13m 16.8M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:50:19.345 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (6m) 5s ago 13m 42.8M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:50:19.345 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (5m) 35s ago 11m 47.7M - 3.5 e1d6a67b021e 8d8009d0c83f 2023-12-16T07:50:19.345 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283,8765 running (5m) 5s ago 19m 435M - 19.0.0-191-g6d933094 a29c258b5d6f 5295ba35c2cd 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (5m) 35s ago 22m 493M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (4m) 35s ago 22m 52.7M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 5a266b45f3ce 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (3m) 5s ago 20m 41.4M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1fc9bf864dda 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (4m) 35s ago 20m 42.0M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1dc91c31fc27 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (6m) 35s ago 14m 13.0M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (6m) 5s ago 14m 16.7M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (2m) 35s ago 19m 52.0M 2782M 19.0.0-191-g6d933094 a29c258b5d6f a00724c6d077 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (104s) 35s ago 18m 51.3M 2782M 19.0.0-191-g6d933094 a29c258b5d6f 5e90fce88079 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (71s) 35s ago 18m 46.9M 2782M 19.0.0-191-g6d933094 a29c258b5d6f 906a30291e88 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (40s) 35s ago 17m - 2782M 19.0.0-191-g6d933094 a29c258b5d6f 8e5e7d6ede0f 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (11s) 5s ago 16m 12.8M 3037M 19.0.0-191-g6d933094 a29c258b5d6f b612e0176579 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (16m) 5s ago 16m 63.9M 3037M 17.2.0 e1d6a67b021e 8b74739ff062 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (15m) 5s ago 15m 64.1M 3037M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (15m) 5s ago 15m 65.0M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:50:19.346 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (5m) 5s ago 14m 63.6M - 2.33.4 514e6a882f6e 2652ce1b8540 2023-12-16T07:50:19.347 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (12m) 35s ago 12m 88.1M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:50:19.347 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (11m) 5s ago 11m 88.4M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:50:19.347 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (11m) 35s ago 11m 88.0M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:50:19.347 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (11m) 5s ago 11m 88.2M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:50:19.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:19 smithi007 bash[64479]: audit 2023-12-16T07:50:18.520557+0000 mgr.y (mgr.25477) 267 : audit 0 from='client.44377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:19.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:19 smithi007 bash[64479]: cluster 2023-12-16T07:50:18.857848+0000 mon.a (mon.0) 444 : cluster 3 Health check update: Degraded data redundancy: 29/630 objects degraded (4.603%), 10 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:19.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:19 smithi007 bash[65864]: audit 2023-12-16T07:50:18.520557+0000 mgr.y (mgr.25477) 267 : audit 0 from='client.44377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:19.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:19 smithi007 bash[65864]: cluster 2023-12-16T07:50:18.857848+0000 mon.a (mon.0) 444 : cluster 3 Health check update: Degraded data redundancy: 29/630 objects degraded (4.603%), 10 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:19.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:19 smithi169 bash[51638]: audit 2023-12-16T07:50:18.520557+0000 mgr.y (mgr.25477) 267 : audit 0 from='client.44377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:19.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:19 smithi169 bash[51638]: cluster 2023-12-16T07:50:18.857848+0000 mon.a (mon.0) 444 : cluster 3 Health check update: Degraded data redundancy: 29/630 objects degraded (4.603%), 10 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:19.807 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:50:19.807 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:50:19.807 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 3 2023-12-16T07:50:19.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:50:19.807 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:50:19.807 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:50:19.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:50:19.807 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:50:19.807 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2023-12-16T07:50:19.807 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 5 2023-12-16T07:50:19.807 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:50:19.807 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:50:19.807 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:50:19.808 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:50:19.808 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:50:19.808 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2023-12-16T07:50:19.808 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 10 2023-12-16T07:50:19.808 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:50:19.808 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:50:20.196 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:50:20.196 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:50:20.196 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:50:20.197 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:50:20.197 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2023-12-16T07:50:20.197 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr", 2023-12-16T07:50:20.197 INFO:teuthology.orchestra.run.smithi007.stdout: "mon" 2023-12-16T07:50:20.197 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:50:20.197 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "10/23 daemons upgraded", 2023-12-16T07:50:20.197 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading osd daemons", 2023-12-16T07:50:20.197 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:50:20.197 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:50:20.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:50:20 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:50:20] "GET /metrics HTTP/1.1" 200 38410 "" "Prometheus/2.33.4" 2023-12-16T07:50:20.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:20 smithi007 bash[65864]: audit 2023-12-16T07:50:18.924703+0000 mgr.y (mgr.25477) 268 : audit 0 from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:20.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:20 smithi007 bash[65864]: cluster 2023-12-16T07:50:19.239022+0000 mgr.y (mgr.25477) 269 : cluster 0 pgmap v161: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 805 B/s rd, 0 op/s 2023-12-16T07:50:20.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:20 smithi007 bash[65864]: audit 2023-12-16T07:50:19.322129+0000 mgr.y (mgr.25477) 270 : audit 0 from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:20.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:20 smithi007 bash[65864]: audit 2023-12-16T07:50:19.803905+0000 mon.c (mon.1) 13 : audit 0 from='client.? 172.21.15.7:0/3018240535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:20.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:20 smithi007 bash[65864]: cluster 2023-12-16T07:50:19.860442+0000 mon.a (mon.0) 445 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/630 objects degraded (4.603%), 10 pgs degraded) 2023-12-16T07:50:20.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:20 smithi007 bash[65864]: cluster 2023-12-16T07:50:19.860514+0000 mon.a (mon.0) 446 : cluster 1 Cluster is now healthy 2023-12-16T07:50:20.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:20 smithi007 bash[64479]: audit 2023-12-16T07:50:18.924703+0000 mgr.y (mgr.25477) 268 : audit 0 from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:20.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:20 smithi007 bash[64479]: cluster 2023-12-16T07:50:19.239022+0000 mgr.y (mgr.25477) 269 : cluster 0 pgmap v161: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 805 B/s rd, 0 op/s 2023-12-16T07:50:20.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:20 smithi007 bash[64479]: audit 2023-12-16T07:50:19.322129+0000 mgr.y (mgr.25477) 270 : audit 0 from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:20.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:20 smithi007 bash[64479]: audit 2023-12-16T07:50:19.803905+0000 mon.c (mon.1) 13 : audit 0 from='client.? 172.21.15.7:0/3018240535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:20.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:20 smithi007 bash[64479]: cluster 2023-12-16T07:50:19.860442+0000 mon.a (mon.0) 445 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/630 objects degraded (4.603%), 10 pgs degraded) 2023-12-16T07:50:20.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:20 smithi007 bash[64479]: cluster 2023-12-16T07:50:19.860514+0000 mon.a (mon.0) 446 : cluster 1 Cluster is now healthy 2023-12-16T07:50:20.658 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:50:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:20 smithi169 bash[51638]: audit 2023-12-16T07:50:18.924703+0000 mgr.y (mgr.25477) 268 : audit 0 from='client.34455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:20 smithi169 bash[51638]: cluster 2023-12-16T07:50:19.239022+0000 mgr.y (mgr.25477) 269 : cluster 0 pgmap v161: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 805 B/s rd, 0 op/s 2023-12-16T07:50:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:20 smithi169 bash[51638]: audit 2023-12-16T07:50:19.322129+0000 mgr.y (mgr.25477) 270 : audit 0 from='client.34461 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:20 smithi169 bash[51638]: audit 2023-12-16T07:50:19.803905+0000 mon.c (mon.1) 13 : audit 0 from='client.? 172.21.15.7:0/3018240535' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:20 smithi169 bash[51638]: cluster 2023-12-16T07:50:19.860442+0000 mon.a (mon.0) 445 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/630 objects degraded (4.603%), 10 pgs degraded) 2023-12-16T07:50:20.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:20 smithi169 bash[51638]: cluster 2023-12-16T07:50:19.860514+0000 mon.a (mon.0) 446 : cluster 1 Cluster is now healthy 2023-12-16T07:50:21.516 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:21 smithi169 bash[51638]: audit 2023-12-16T07:50:20.193002+0000 mgr.y (mgr.25477) 271 : audit 0 from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:21.516 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:21 smithi169 bash[51638]: audit 2023-12-16T07:50:20.655854+0000 mon.a (mon.0) 447 : audit 0 from='client.? 172.21.15.7:0/74655672' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:50:21.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:21 smithi007 bash[64479]: audit 2023-12-16T07:50:20.193002+0000 mgr.y (mgr.25477) 271 : audit 0 from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:21.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:21 smithi007 bash[64479]: audit 2023-12-16T07:50:20.655854+0000 mon.a (mon.0) 447 : audit 0 from='client.? 172.21.15.7:0/74655672' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:50:21.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:21 smithi007 bash[65864]: audit 2023-12-16T07:50:20.193002+0000 mgr.y (mgr.25477) 271 : audit 0 from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:21.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:21 smithi007 bash[65864]: audit 2023-12-16T07:50:20.655854+0000 mon.a (mon.0) 447 : audit 0 from='client.? 172.21.15.7:0/74655672' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:50:22.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:22 smithi169 bash[51638]: cluster 2023-12-16T07:50:21.240277+0000 mgr.y (mgr.25477) 272 : cluster 0 pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:50:22.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:22 smithi007 bash[65864]: cluster 2023-12-16T07:50:21.240277+0000 mgr.y (mgr.25477) 272 : cluster 0 pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:50:22.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:22 smithi007 bash[64479]: cluster 2023-12-16T07:50:21.240277+0000 mgr.y (mgr.25477) 272 : cluster 0 pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:50:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:24 smithi169 bash[51638]: audit 2023-12-16T07:50:23.211750+0000 mon.a (mon.0) 448 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:24 smithi169 bash[51638]: audit 2023-12-16T07:50:23.214223+0000 mon.a (mon.0) 449 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:50:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:24 smithi169 bash[51638]: cluster 2023-12-16T07:50:23.241971+0000 mgr.y (mgr.25477) 273 : cluster 0 pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:50:24.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:24 smithi007 bash[64479]: audit 2023-12-16T07:50:23.211750+0000 mon.a (mon.0) 448 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:24.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:24 smithi007 bash[64479]: audit 2023-12-16T07:50:23.214223+0000 mon.a (mon.0) 449 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:50:24.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:24 smithi007 bash[64479]: cluster 2023-12-16T07:50:23.241971+0000 mgr.y (mgr.25477) 273 : cluster 0 pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:50:24.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:24 smithi007 bash[65864]: audit 2023-12-16T07:50:23.211750+0000 mon.a (mon.0) 448 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:24.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:24 smithi007 bash[65864]: audit 2023-12-16T07:50:23.214223+0000 mon.a (mon.0) 449 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:50:24.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:24 smithi007 bash[65864]: cluster 2023-12-16T07:50:23.241971+0000 mgr.y (mgr.25477) 273 : cluster 0 pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:50:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[64479]: audit 2023-12-16T07:50:24.319223+0000 mgr.y (mgr.25477) 274 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[64479]: audit 2023-12-16T07:50:24.808007+0000 mon.a (mon.0) 450 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[64479]: audit 2023-12-16T07:50:24.819241+0000 mon.a (mon.0) 451 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[64479]: audit 2023-12-16T07:50:24.821506+0000 mon.a (mon.0) 452 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:50:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[64479]: audit 2023-12-16T07:50:24.823333+0000 mon.a (mon.0) 453 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:50:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[64479]: audit 2023-12-16T07:50:24.832651+0000 mon.a (mon.0) 454 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:25.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[64479]: audit 2023-12-16T07:50:24.921114+0000 mon.a (mon.0) 455 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:50:25.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[64479]: audit 2023-12-16T07:50:24.925065+0000 mon.a (mon.0) 456 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:25.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[64479]: audit 2023-12-16T07:50:24.928283+0000 mon.a (mon.0) 457 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:25.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[64479]: audit 2023-12-16T07:50:24.930672+0000 mon.a (mon.0) 458 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:25.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[64479]: audit 2023-12-16T07:50:24.933523+0000 mon.a (mon.0) 459 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-12-16T07:50:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[65864]: audit 2023-12-16T07:50:24.319223+0000 mgr.y (mgr.25477) 274 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[65864]: audit 2023-12-16T07:50:24.808007+0000 mon.a (mon.0) 450 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[65864]: audit 2023-12-16T07:50:24.819241+0000 mon.a (mon.0) 451 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[65864]: audit 2023-12-16T07:50:24.821506+0000 mon.a (mon.0) 452 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:50:25.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[65864]: audit 2023-12-16T07:50:24.823333+0000 mon.a (mon.0) 453 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:50:25.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[65864]: audit 2023-12-16T07:50:24.832651+0000 mon.a (mon.0) 454 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:25.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[65864]: audit 2023-12-16T07:50:24.921114+0000 mon.a (mon.0) 455 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:50:25.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[65864]: audit 2023-12-16T07:50:24.925065+0000 mon.a (mon.0) 456 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:25.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[65864]: audit 2023-12-16T07:50:24.928283+0000 mon.a (mon.0) 457 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:25.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[65864]: audit 2023-12-16T07:50:24.930672+0000 mon.a (mon.0) 458 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:25.570 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:25 smithi007 bash[65864]: audit 2023-12-16T07:50:24.933523+0000 mon.a (mon.0) 459 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-12-16T07:50:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:25 smithi169 bash[51638]: audit 2023-12-16T07:50:24.319223+0000 mgr.y (mgr.25477) 274 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:25 smithi169 bash[51638]: audit 2023-12-16T07:50:24.808007+0000 mon.a (mon.0) 450 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:25 smithi169 bash[51638]: audit 2023-12-16T07:50:24.819241+0000 mon.a (mon.0) 451 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:25 smithi169 bash[51638]: audit 2023-12-16T07:50:24.821506+0000 mon.a (mon.0) 452 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:50:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:25 smithi169 bash[51638]: audit 2023-12-16T07:50:24.823333+0000 mon.a (mon.0) 453 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:50:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:25 smithi169 bash[51638]: audit 2023-12-16T07:50:24.832651+0000 mon.a (mon.0) 454 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:25 smithi169 bash[51638]: audit 2023-12-16T07:50:24.921114+0000 mon.a (mon.0) 455 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:50:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:25 smithi169 bash[51638]: audit 2023-12-16T07:50:24.925065+0000 mon.a (mon.0) 456 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:25 smithi169 bash[51638]: audit 2023-12-16T07:50:24.928283+0000 mon.a (mon.0) 457 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:25 smithi169 bash[51638]: audit 2023-12-16T07:50:24.930672+0000 mon.a (mon.0) 458 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:25 smithi169 bash[51638]: audit 2023-12-16T07:50:24.933523+0000 mon.a (mon.0) 459 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-12-16T07:50:26.307 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:26 smithi169 bash[51638]: audit 2023-12-16T07:50:24.933953+0000 mgr.y (mgr.25477) 275 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-12-16T07:50:26.307 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:26 smithi169 bash[51638]: cephadm 2023-12-16T07:50:24.936367+0000 mgr.y (mgr.25477) 276 : cephadm 1 Upgrade: osd.5 is safe to restart 2023-12-16T07:50:26.307 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:26 smithi169 bash[51638]: cluster 2023-12-16T07:50:25.243498+0000 mgr.y (mgr.25477) 277 : cluster 0 pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2023-12-16T07:50:26.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:26 smithi007 bash[65864]: audit 2023-12-16T07:50:24.933953+0000 mgr.y (mgr.25477) 275 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-12-16T07:50:26.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:26 smithi007 bash[65864]: cephadm 2023-12-16T07:50:24.936367+0000 mgr.y (mgr.25477) 276 : cephadm 1 Upgrade: osd.5 is safe to restart 2023-12-16T07:50:26.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:26 smithi007 bash[65864]: cluster 2023-12-16T07:50:25.243498+0000 mgr.y (mgr.25477) 277 : cluster 0 pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2023-12-16T07:50:26.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:26 smithi007 bash[64479]: audit 2023-12-16T07:50:24.933953+0000 mgr.y (mgr.25477) 275 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2023-12-16T07:50:26.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:26 smithi007 bash[64479]: cephadm 2023-12-16T07:50:24.936367+0000 mgr.y (mgr.25477) 276 : cephadm 1 Upgrade: osd.5 is safe to restart 2023-12-16T07:50:26.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:26 smithi007 bash[64479]: cluster 2023-12-16T07:50:25.243498+0000 mgr.y (mgr.25477) 277 : cluster 0 pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 989 B/s rd, 0 op/s 2023-12-16T07:50:28.305 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:28 smithi169 bash[51638]: cephadm 2023-12-16T07:50:27.006562+0000 mgr.y (mgr.25477) 278 : cephadm 1 Upgrade: Updating osd.5 2023-12-16T07:50:28.305 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:28 smithi169 bash[51638]: audit 2023-12-16T07:50:27.013891+0000 mon.a (mon.0) 460 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:28.305 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:28 smithi169 bash[51638]: audit 2023-12-16T07:50:27.018422+0000 mon.a (mon.0) 461 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-16T07:50:28.306 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:28 smithi169 bash[51638]: audit 2023-12-16T07:50:27.019568+0000 mon.a (mon.0) 462 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:50:28.306 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:28 smithi169 bash[51638]: cephadm 2023-12-16T07:50:27.024955+0000 mgr.y (mgr.25477) 279 : cephadm 1 Deploying daemon osd.5 on smithi169 2023-12-16T07:50:28.306 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:28 smithi169 bash[51638]: cluster 2023-12-16T07:50:27.244411+0000 mgr.y (mgr.25477) 280 : cluster 0 pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:50:28.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:28 smithi007 bash[64479]: cephadm 2023-12-16T07:50:27.006562+0000 mgr.y (mgr.25477) 278 : cephadm 1 Upgrade: Updating osd.5 2023-12-16T07:50:28.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:28 smithi007 bash[64479]: audit 2023-12-16T07:50:27.013891+0000 mon.a (mon.0) 460 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:28.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:28 smithi007 bash[64479]: audit 2023-12-16T07:50:27.018422+0000 mon.a (mon.0) 461 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-16T07:50:28.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:28 smithi007 bash[64479]: audit 2023-12-16T07:50:27.019568+0000 mon.a (mon.0) 462 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:50:28.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:28 smithi007 bash[64479]: cephadm 2023-12-16T07:50:27.024955+0000 mgr.y (mgr.25477) 279 : cephadm 1 Deploying daemon osd.5 on smithi169 2023-12-16T07:50:28.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:28 smithi007 bash[64479]: cluster 2023-12-16T07:50:27.244411+0000 mgr.y (mgr.25477) 280 : cluster 0 pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:50:28.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:28 smithi007 bash[65864]: cephadm 2023-12-16T07:50:27.006562+0000 mgr.y (mgr.25477) 278 : cephadm 1 Upgrade: Updating osd.5 2023-12-16T07:50:28.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:28 smithi007 bash[65864]: audit 2023-12-16T07:50:27.013891+0000 mon.a (mon.0) 460 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:28.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:28 smithi007 bash[65864]: audit 2023-12-16T07:50:27.018422+0000 mon.a (mon.0) 461 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-12-16T07:50:28.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:28 smithi007 bash[65864]: audit 2023-12-16T07:50:27.019568+0000 mon.a (mon.0) 462 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:50:28.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:28 smithi007 bash[65864]: cephadm 2023-12-16T07:50:27.024955+0000 mgr.y (mgr.25477) 279 : cephadm 1 Deploying daemon osd.5 on smithi169 2023-12-16T07:50:28.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:28 smithi007 bash[65864]: cluster 2023-12-16T07:50:27.244411+0000 mgr.y (mgr.25477) 280 : cluster 0 pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:50:29.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:29 smithi007 bash[64479]: audit 2023-12-16T07:50:28.453235+0000 mon.a (mon.0) 463 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:29.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:29 smithi007 bash[65864]: audit 2023-12-16T07:50:28.453235+0000 mon.a (mon.0) 463 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:29.888 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:29 smithi169 bash[51638]: audit 2023-12-16T07:50:28.453235+0000 mon.a (mon.0) 463 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:30.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:50:30 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:50:30] "GET /metrics HTTP/1.1" 200 38414 "" "Prometheus/2.33.4" 2023-12-16T07:50:30.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:30 smithi007 bash[65864]: cluster 2023-12-16T07:50:29.245717+0000 mgr.y (mgr.25477) 281 : cluster 0 pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:50:30.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:30 smithi007 bash[64479]: cluster 2023-12-16T07:50:29.245717+0000 mgr.y (mgr.25477) 281 : cluster 0 pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:50:30.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:30 smithi169 bash[51638]: cluster 2023-12-16T07:50:29.245717+0000 mgr.y (mgr.25477) 281 : cluster 0 pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:50:31.433 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:31 smithi169 systemd[1]: Stopping Ceph osd.5 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:50:31.933 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:31 smithi169 bash[31281]: debug 2023-12-16T07:50:31.495+0000 7fbd0cd6b700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:50:31.933 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:31 smithi169 bash[31281]: debug 2023-12-16T07:50:31.495+0000 7fbd0cd6b700 -1 osd.5 128 *** Got signal Terminated *** 2023-12-16T07:50:31.933 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:31 smithi169 bash[31281]: debug 2023-12-16T07:50:31.495+0000 7fbd0cd6b700 -1 osd.5 128 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:50:32.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:32 smithi007 bash[64479]: cluster 2023-12-16T07:50:31.246963+0000 mgr.y (mgr.25477) 282 : cluster 0 pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:50:32.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:32 smithi007 bash[64479]: cluster 2023-12-16T07:50:31.499149+0000 mon.a (mon.0) 464 : cluster 1 osd.5 marked itself down and dead 2023-12-16T07:50:32.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:32 smithi007 bash[65864]: cluster 2023-12-16T07:50:31.246963+0000 mgr.y (mgr.25477) 282 : cluster 0 pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:50:32.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:32 smithi007 bash[65864]: cluster 2023-12-16T07:50:31.499149+0000 mon.a (mon.0) 464 : cluster 1 osd.5 marked itself down and dead 2023-12-16T07:50:32.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:32 smithi169 bash[51638]: cluster 2023-12-16T07:50:31.246963+0000 mgr.y (mgr.25477) 282 : cluster 0 pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:50:32.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:32 smithi169 bash[51638]: cluster 2023-12-16T07:50:31.499149+0000 mon.a (mon.0) 464 : cluster 1 osd.5 marked itself down and dead 2023-12-16T07:50:33.750 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:33 smithi169 bash[55512]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-5 2023-12-16T07:50:33.750 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:33 smithi169 bash[51638]: cluster 2023-12-16T07:50:32.457005+0000 mon.a (mon.0) 465 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:50:33.751 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:33 smithi169 bash[51638]: cluster 2023-12-16T07:50:32.471284+0000 osd.6 (osd.6) 37 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.751 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:33 smithi169 bash[51638]: cluster 2023-12-16T07:50:32.474284+0000 mon.a (mon.0) 466 : cluster 0 osdmap e129: 8 total, 7 up, 8 in 2023-12-16T07:50:33.751 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:33 smithi169 bash[51638]: cluster 2023-12-16T07:50:32.475961+0000 osd.7 (osd.7) 36 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.751 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:33 smithi169 bash[51638]: cluster 2023-12-16T07:50:32.478235+0000 osd.6 (osd.6) 38 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.751 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:33 smithi169 bash[51638]: cluster 2023-12-16T07:50:32.479763+0000 osd.7 (osd.7) 37 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.751 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:33 smithi169 bash[51638]: cluster 2023-12-16T07:50:32.481847+0000 osd.6 (osd.6) 39 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.751 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:33 smithi169 bash[51638]: cluster 2023-12-16T07:50:32.482411+0000 osd.7 (osd.7) 38 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.751 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:33 smithi169 bash[51638]: cluster 2023-12-16T07:50:32.483078+0000 osd.6 (osd.6) 40 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.751 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:33 smithi169 bash[51638]: cluster 2023-12-16T07:50:32.486394+0000 osd.6 (osd.6) 41 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[64479]: cluster 2023-12-16T07:50:32.457005+0000 mon.a (mon.0) 465 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:50:33.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[64479]: cluster 2023-12-16T07:50:32.471284+0000 osd.6 (osd.6) 37 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[64479]: cluster 2023-12-16T07:50:32.474284+0000 mon.a (mon.0) 466 : cluster 0 osdmap e129: 8 total, 7 up, 8 in 2023-12-16T07:50:33.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[64479]: cluster 2023-12-16T07:50:32.475961+0000 osd.7 (osd.7) 36 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[64479]: cluster 2023-12-16T07:50:32.478235+0000 osd.6 (osd.6) 38 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[64479]: cluster 2023-12-16T07:50:32.479763+0000 osd.7 (osd.7) 37 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[64479]: cluster 2023-12-16T07:50:32.481847+0000 osd.6 (osd.6) 39 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[64479]: cluster 2023-12-16T07:50:32.482411+0000 osd.7 (osd.7) 38 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[64479]: cluster 2023-12-16T07:50:32.483078+0000 osd.6 (osd.6) 40 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[64479]: cluster 2023-12-16T07:50:32.486394+0000 osd.6 (osd.6) 41 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[65864]: cluster 2023-12-16T07:50:32.457005+0000 mon.a (mon.0) 465 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:50:33.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[65864]: cluster 2023-12-16T07:50:32.471284+0000 osd.6 (osd.6) 37 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[65864]: cluster 2023-12-16T07:50:32.474284+0000 mon.a (mon.0) 466 : cluster 0 osdmap e129: 8 total, 7 up, 8 in 2023-12-16T07:50:33.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[65864]: cluster 2023-12-16T07:50:32.475961+0000 osd.7 (osd.7) 36 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[65864]: cluster 2023-12-16T07:50:32.478235+0000 osd.6 (osd.6) 38 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[65864]: cluster 2023-12-16T07:50:32.479763+0000 osd.7 (osd.7) 37 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[65864]: cluster 2023-12-16T07:50:32.481847+0000 osd.6 (osd.6) 39 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[65864]: cluster 2023-12-16T07:50:32.482411+0000 osd.7 (osd.7) 38 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[65864]: cluster 2023-12-16T07:50:32.483078+0000 osd.6 (osd.6) 40 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:33.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:33 smithi007 bash[65864]: cluster 2023-12-16T07:50:32.486394+0000 osd.6 (osd.6) 41 : cluster 3 failed to encode map e129 with expected crc 2023-12-16T07:50:34.183 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:33 smithi169 bash[55586]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-5 2023-12-16T07:50:34.756 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:34 smithi169 bash[51638]: cluster 2023-12-16T07:50:33.247942+0000 mgr.y (mgr.25477) 283 : cluster 0 pgmap v169: 161 pgs: 9 peering, 17 stale+active+clean, 135 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2023-12-16T07:50:34.756 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:34 smithi169 bash[51638]: cluster 2023-12-16T07:50:33.468041+0000 mon.a (mon.0) 467 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-12-16T07:50:34.756 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:34 smithi169 bash[51638]: cluster 2023-12-16T07:50:33.477572+0000 mon.a (mon.0) 468 : cluster 0 osdmap e130: 8 total, 7 up, 8 in 2023-12-16T07:50:34.756 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:34 smithi169 bash[51638]: cluster 2023-12-16T07:50:33.477745+0000 osd.6 (osd.6) 42 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.756 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:34 smithi169 bash[51638]: cluster 2023-12-16T07:50:33.478374+0000 osd.7 (osd.7) 39 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.756 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:34 smithi169 bash[51638]: cluster 2023-12-16T07:50:33.482025+0000 osd.7 (osd.7) 40 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.756 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:34 smithi169 bash[51638]: cluster 2023-12-16T07:50:33.482493+0000 osd.6 (osd.6) 43 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.756 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:34 smithi169 bash[51638]: cluster 2023-12-16T07:50:33.485759+0000 osd.6 (osd.6) 44 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.756 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:34 smithi169 bash[51638]: cluster 2023-12-16T07:50:33.486165+0000 osd.7 (osd.7) 41 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[65864]: cluster 2023-12-16T07:50:33.247942+0000 mgr.y (mgr.25477) 283 : cluster 0 pgmap v169: 161 pgs: 9 peering, 17 stale+active+clean, 135 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2023-12-16T07:50:34.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[65864]: cluster 2023-12-16T07:50:33.468041+0000 mon.a (mon.0) 467 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-12-16T07:50:34.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[65864]: cluster 2023-12-16T07:50:33.477572+0000 mon.a (mon.0) 468 : cluster 0 osdmap e130: 8 total, 7 up, 8 in 2023-12-16T07:50:34.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[65864]: cluster 2023-12-16T07:50:33.477745+0000 osd.6 (osd.6) 42 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[65864]: cluster 2023-12-16T07:50:33.478374+0000 osd.7 (osd.7) 39 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[65864]: cluster 2023-12-16T07:50:33.482025+0000 osd.7 (osd.7) 40 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[65864]: cluster 2023-12-16T07:50:33.482493+0000 osd.6 (osd.6) 43 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[65864]: cluster 2023-12-16T07:50:33.485759+0000 osd.6 (osd.6) 44 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[65864]: cluster 2023-12-16T07:50:33.486165+0000 osd.7 (osd.7) 41 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[64479]: cluster 2023-12-16T07:50:33.247942+0000 mgr.y (mgr.25477) 283 : cluster 0 pgmap v169: 161 pgs: 9 peering, 17 stale+active+clean, 135 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2023-12-16T07:50:34.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[64479]: cluster 2023-12-16T07:50:33.468041+0000 mon.a (mon.0) 467 : cluster 3 Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-12-16T07:50:34.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[64479]: cluster 2023-12-16T07:50:33.477572+0000 mon.a (mon.0) 468 : cluster 0 osdmap e130: 8 total, 7 up, 8 in 2023-12-16T07:50:34.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[64479]: cluster 2023-12-16T07:50:33.477745+0000 osd.6 (osd.6) 42 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[64479]: cluster 2023-12-16T07:50:33.478374+0000 osd.7 (osd.7) 39 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[64479]: cluster 2023-12-16T07:50:33.482025+0000 osd.7 (osd.7) 40 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[64479]: cluster 2023-12-16T07:50:33.482493+0000 osd.6 (osd.6) 43 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[64479]: cluster 2023-12-16T07:50:33.485759+0000 osd.6 (osd.6) 44 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:34.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:34 smithi007 bash[64479]: cluster 2023-12-16T07:50:33.486165+0000 osd.7 (osd.7) 41 : cluster 3 failed to encode map e130 with expected crc 2023-12-16T07:50:35.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:35 smithi169 bash[51638]: audit 2023-12-16T07:50:34.327110+0000 mgr.y (mgr.25477) 284 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:35.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:35 smithi007 bash[65864]: audit 2023-12-16T07:50:34.327110+0000 mgr.y (mgr.25477) 284 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:35.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:35 smithi007 bash[64479]: audit 2023-12-16T07:50:34.327110+0000 mgr.y (mgr.25477) 284 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:36.090 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:35 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.5.service: Succeeded. 2023-12-16T07:50:36.090 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:35 smithi169 systemd[1]: Stopped Ceph osd.5 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:50:36.091 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:36 smithi169 systemd[1]: Started Ceph osd.5 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:50:36.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:36 smithi007 bash[65864]: cluster 2023-12-16T07:50:35.249589+0000 mgr.y (mgr.25477) 285 : cluster 0 pgmap v171: 161 pgs: 14 active+undersized+degraded, 18 active+undersized, 14 peering, 5 stale+active+clean, 110 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 46/630 objects degraded (7.302%) 2023-12-16T07:50:36.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:36 smithi007 bash[65864]: cluster 2023-12-16T07:50:35.462677+0000 mon.a (mon.0) 469 : cluster 3 Health check failed: Degraded data redundancy: 46/630 objects degraded (7.302%), 14 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:36.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:36 smithi007 bash[65864]: audit 2023-12-16T07:50:36.081200+0000 mon.a (mon.0) 470 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:36.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:36 smithi007 bash[65864]: audit 2023-12-16T07:50:36.092010+0000 mon.a (mon.0) 471 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:36.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:36 smithi007 bash[64479]: cluster 2023-12-16T07:50:35.249589+0000 mgr.y (mgr.25477) 285 : cluster 0 pgmap v171: 161 pgs: 14 active+undersized+degraded, 18 active+undersized, 14 peering, 5 stale+active+clean, 110 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 46/630 objects degraded (7.302%) 2023-12-16T07:50:36.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:36 smithi007 bash[64479]: cluster 2023-12-16T07:50:35.462677+0000 mon.a (mon.0) 469 : cluster 3 Health check failed: Degraded data redundancy: 46/630 objects degraded (7.302%), 14 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:36.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:36 smithi007 bash[64479]: audit 2023-12-16T07:50:36.081200+0000 mon.a (mon.0) 470 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:36.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:36 smithi007 bash[64479]: audit 2023-12-16T07:50:36.092010+0000 mon.a (mon.0) 471 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:36.905 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:36 smithi169 bash[51638]: cluster 2023-12-16T07:50:35.249589+0000 mgr.y (mgr.25477) 285 : cluster 0 pgmap v171: 161 pgs: 14 active+undersized+degraded, 18 active+undersized, 14 peering, 5 stale+active+clean, 110 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 46/630 objects degraded (7.302%) 2023-12-16T07:50:36.905 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:36 smithi169 bash[51638]: cluster 2023-12-16T07:50:35.462677+0000 mon.a (mon.0) 469 : cluster 3 Health check failed: Degraded data redundancy: 46/630 objects degraded (7.302%), 14 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:36.905 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:36 smithi169 bash[51638]: audit 2023-12-16T07:50:36.081200+0000 mon.a (mon.0) 470 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:36.905 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:36 smithi169 bash[51638]: audit 2023-12-16T07:50:36.092010+0000 mon.a (mon.0) 471 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:37.933 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:37 smithi169 bash[55756]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-16T07:50:37.933 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:37 smithi169 bash[55756]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2023-12-16T07:50:37.933 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:37 smithi169 bash[55756]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2023-12-16T07:50:37.933 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:37 smithi169 bash[55756]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2023-12-16T07:50:37.933 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:37 smithi169 bash[55756]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2023-12-16T07:50:37.933 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:37 smithi169 bash[55756]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-16T07:50:37.933 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:37 smithi169 bash[55756]: --> ceph-volume raw activate successful for osd ID: 5 2023-12-16T07:50:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:38 smithi169 bash[51638]: cluster 2023-12-16T07:50:37.250864+0000 mgr.y (mgr.25477) 286 : cluster 0 pgmap v172: 161 pgs: 15 active+undersized+degraded, 30 active+undersized, 14 peering, 102 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 56/630 objects degraded (8.889%) 2023-12-16T07:50:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:38 smithi169 bash[51638]: audit 2023-12-16T07:50:38.212693+0000 mon.a (mon.0) 472 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:38 smithi169 bash[51638]: audit 2023-12-16T07:50:38.214884+0000 mon.a (mon.0) 473 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:50:38.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:38 smithi007 bash[65864]: cluster 2023-12-16T07:50:37.250864+0000 mgr.y (mgr.25477) 286 : cluster 0 pgmap v172: 161 pgs: 15 active+undersized+degraded, 30 active+undersized, 14 peering, 102 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 56/630 objects degraded (8.889%) 2023-12-16T07:50:38.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:38 smithi007 bash[65864]: audit 2023-12-16T07:50:38.212693+0000 mon.a (mon.0) 472 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:38.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:38 smithi007 bash[65864]: audit 2023-12-16T07:50:38.214884+0000 mon.a (mon.0) 473 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:50:38.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:38 smithi007 bash[64479]: cluster 2023-12-16T07:50:37.250864+0000 mgr.y (mgr.25477) 286 : cluster 0 pgmap v172: 161 pgs: 15 active+undersized+degraded, 30 active+undersized, 14 peering, 102 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 56/630 objects degraded (8.889%) 2023-12-16T07:50:38.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:38 smithi007 bash[64479]: audit 2023-12-16T07:50:38.212693+0000 mon.a (mon.0) 472 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:38.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:38 smithi007 bash[64479]: audit 2023-12-16T07:50:38.214884+0000 mon.a (mon.0) 473 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:50:39.433 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:39 smithi169 bash[56063]: debug 2023-12-16T07:50:39.039+0000 7f0da01ca700 1 -- 172.21.15.169:0/1060798731 <== mon.2 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564f8eacf6c0 con 0x564f8eb55000 2023-12-16T07:50:39.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:39 smithi007 bash[64479]: cluster 2023-12-16T07:50:38.864966+0000 mon.a (mon.0) 474 : cluster 3 Health check update: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2023-12-16T07:50:39.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:39 smithi007 bash[65864]: cluster 2023-12-16T07:50:38.864966+0000 mon.a (mon.0) 474 : cluster 3 Health check update: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2023-12-16T07:50:39.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:39 smithi169 bash[51638]: cluster 2023-12-16T07:50:38.864966+0000 mon.a (mon.0) 474 : cluster 3 Health check update: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2023-12-16T07:50:39.934 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:39 smithi169 bash[56063]: debug 2023-12-16T07:50:39.607+0000 7f0da9350640 -1 Falling back to public interface 2023-12-16T07:50:40.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:50:40 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:50:40] "GET /metrics HTTP/1.1" 200 38414 "" "Prometheus/2.33.4" 2023-12-16T07:50:40.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:40 smithi007 bash[64479]: cluster 2023-12-16T07:50:39.252322+0000 mgr.y (mgr.25477) 287 : cluster 0 pgmap v173: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2023-12-16T07:50:40.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:40 smithi007 bash[64479]: cluster 2023-12-16T07:50:39.868975+0000 mon.a (mon.0) 475 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2023-12-16T07:50:40.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:40 smithi007 bash[65864]: cluster 2023-12-16T07:50:39.252322+0000 mgr.y (mgr.25477) 287 : cluster 0 pgmap v173: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2023-12-16T07:50:40.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:40 smithi007 bash[65864]: cluster 2023-12-16T07:50:39.868975+0000 mon.a (mon.0) 475 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2023-12-16T07:50:40.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:40 smithi169 bash[51638]: cluster 2023-12-16T07:50:39.252322+0000 mgr.y (mgr.25477) 287 : cluster 0 pgmap v173: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2023-12-16T07:50:40.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:40 smithi169 bash[51638]: cluster 2023-12-16T07:50:39.868975+0000 mon.a (mon.0) 475 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2023-12-16T07:50:42.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:42 smithi007 bash[65864]: cluster 2023-12-16T07:50:41.253907+0000 mgr.y (mgr.25477) 288 : cluster 0 pgmap v174: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2023-12-16T07:50:42.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:42 smithi007 bash[65864]: audit 2023-12-16T07:50:42.466653+0000 mon.a (mon.0) 476 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:42.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:42 smithi007 bash[65864]: audit 2023-12-16T07:50:42.476904+0000 mon.a (mon.0) 477 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:42.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:42 smithi007 bash[64479]: cluster 2023-12-16T07:50:41.253907+0000 mgr.y (mgr.25477) 288 : cluster 0 pgmap v174: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2023-12-16T07:50:42.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:42 smithi007 bash[64479]: audit 2023-12-16T07:50:42.466653+0000 mon.a (mon.0) 476 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:42.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:42 smithi007 bash[64479]: audit 2023-12-16T07:50:42.476904+0000 mon.a (mon.0) 477 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:42.933 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:42 smithi169 bash[56063]: debug 2023-12-16T07:50:42.611+0000 7f0da9350640 -1 osd.5 0 read_superblock omap replica is missing. 2023-12-16T07:50:42.933 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:42 smithi169 bash[56063]: debug 2023-12-16T07:50:42.655+0000 7f0da9350640 -1 osd.5 128 log_to_monitors true 2023-12-16T07:50:42.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:42 smithi169 bash[51638]: cluster 2023-12-16T07:50:41.253907+0000 mgr.y (mgr.25477) 288 : cluster 0 pgmap v174: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2023-12-16T07:50:42.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:42 smithi169 bash[51638]: audit 2023-12-16T07:50:42.466653+0000 mon.a (mon.0) 476 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:42.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:42 smithi169 bash[51638]: audit 2023-12-16T07:50:42.476904+0000 mon.a (mon.0) 477 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:43.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:43 smithi007 bash[64479]: audit 2023-12-16T07:50:42.663091+0000 mon.a (mon.0) 478 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T07:50:43.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:43 smithi007 bash[64479]: audit 2023-12-16T07:50:42.664573+0000 mon.b (mon.2) 5 : audit 1 from='osd.5 [v2:172.21.15.169:6808/2407091839,v1:172.21.15.169:6809/2407091839]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T07:50:43.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:43 smithi007 bash[65864]: audit 2023-12-16T07:50:42.663091+0000 mon.a (mon.0) 478 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T07:50:43.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:43 smithi007 bash[65864]: audit 2023-12-16T07:50:42.664573+0000 mon.b (mon.2) 5 : audit 1 from='osd.5 [v2:172.21.15.169:6808/2407091839,v1:172.21.15.169:6809/2407091839]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T07:50:43.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:43 smithi169 bash[51638]: audit 2023-12-16T07:50:42.663091+0000 mon.a (mon.0) 478 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T07:50:43.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:43 smithi169 bash[51638]: audit 2023-12-16T07:50:42.664573+0000 mon.b (mon.2) 5 : audit 1 from='osd.5 [v2:172.21.15.169:6808/2407091839,v1:172.21.15.169:6809/2407091839]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-12-16T07:50:43.933 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:50:43 smithi169 bash[56063]: debug 2023-12-16T07:50:43.539+0000 7f0d97fbc700 -1 osd.5 128 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:50:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[64479]: cluster 2023-12-16T07:50:43.255313+0000 mgr.y (mgr.25477) 289 : cluster 0 pgmap v175: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2023-12-16T07:50:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[64479]: audit 2023-12-16T07:50:43.497419+0000 mon.a (mon.0) 479 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-16T07:50:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[64479]: cluster 2023-12-16T07:50:43.501435+0000 osd.6 (osd.6) 45 : cluster 3 failed to encode map e131 with expected crc 2023-12-16T07:50:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[64479]: cluster 2023-12-16T07:50:43.501624+0000 mon.a (mon.0) 480 : cluster 0 osdmap e131: 8 total, 7 up, 8 in 2023-12-16T07:50:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[64479]: audit 2023-12-16T07:50:43.505661+0000 mon.a (mon.0) 481 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:50:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[64479]: audit 2023-12-16T07:50:43.507222+0000 mon.b (mon.2) 6 : audit 1 from='osd.5 [v2:172.21.15.169:6808/2407091839,v1:172.21.15.169:6809/2407091839]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:50:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[64479]: audit 2023-12-16T07:50:43.578796+0000 mon.a (mon.0) 482 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[64479]: audit 2023-12-16T07:50:43.589498+0000 mon.a (mon.0) 483 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[64479]: cluster 2023-12-16T07:50:43.867261+0000 mon.a (mon.0) 484 : cluster 3 Health check update: Degraded data redundancy: 71/630 objects degraded (11.270%), 19 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:44.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[65864]: cluster 2023-12-16T07:50:43.255313+0000 mgr.y (mgr.25477) 289 : cluster 0 pgmap v175: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2023-12-16T07:50:44.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[65864]: audit 2023-12-16T07:50:43.497419+0000 mon.a (mon.0) 479 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-16T07:50:44.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[65864]: cluster 2023-12-16T07:50:43.501435+0000 osd.6 (osd.6) 45 : cluster 3 failed to encode map e131 with expected crc 2023-12-16T07:50:44.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[65864]: cluster 2023-12-16T07:50:43.501624+0000 mon.a (mon.0) 480 : cluster 0 osdmap e131: 8 total, 7 up, 8 in 2023-12-16T07:50:44.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[65864]: audit 2023-12-16T07:50:43.505661+0000 mon.a (mon.0) 481 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:50:44.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[65864]: audit 2023-12-16T07:50:43.507222+0000 mon.b (mon.2) 6 : audit 1 from='osd.5 [v2:172.21.15.169:6808/2407091839,v1:172.21.15.169:6809/2407091839]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:50:44.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[65864]: audit 2023-12-16T07:50:43.578796+0000 mon.a (mon.0) 482 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:44.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[65864]: audit 2023-12-16T07:50:43.589498+0000 mon.a (mon.0) 483 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:44.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:44 smithi007 bash[65864]: cluster 2023-12-16T07:50:43.867261+0000 mon.a (mon.0) 484 : cluster 3 Health check update: Degraded data redundancy: 71/630 objects degraded (11.270%), 19 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:44 smithi169 bash[51638]: cluster 2023-12-16T07:50:43.255313+0000 mgr.y (mgr.25477) 289 : cluster 0 pgmap v175: 161 pgs: 19 active+undersized+degraded, 40 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 71/630 objects degraded (11.270%) 2023-12-16T07:50:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:44 smithi169 bash[51638]: audit 2023-12-16T07:50:43.497419+0000 mon.a (mon.0) 479 : audit 1 from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-12-16T07:50:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:44 smithi169 bash[51638]: cluster 2023-12-16T07:50:43.501435+0000 osd.6 (osd.6) 45 : cluster 3 failed to encode map e131 with expected crc 2023-12-16T07:50:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:44 smithi169 bash[51638]: cluster 2023-12-16T07:50:43.501624+0000 mon.a (mon.0) 480 : cluster 0 osdmap e131: 8 total, 7 up, 8 in 2023-12-16T07:50:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:44 smithi169 bash[51638]: audit 2023-12-16T07:50:43.505661+0000 mon.a (mon.0) 481 : audit 1 from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:50:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:44 smithi169 bash[51638]: audit 2023-12-16T07:50:43.507222+0000 mon.b (mon.2) 6 : audit 1 from='osd.5 [v2:172.21.15.169:6808/2407091839,v1:172.21.15.169:6809/2407091839]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:50:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:44 smithi169 bash[51638]: audit 2023-12-16T07:50:43.578796+0000 mon.a (mon.0) 482 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:44 smithi169 bash[51638]: audit 2023-12-16T07:50:43.589498+0000 mon.a (mon.0) 483 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:44.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:44 smithi169 bash[51638]: cluster 2023-12-16T07:50:43.867261+0000 mon.a (mon.0) 484 : cluster 3 Health check update: Degraded data redundancy: 71/630 objects degraded (11.270%), 19 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[64479]: cluster 2023-12-16T07:50:44.187238+0000 osd.7 (osd.7) 42 : cluster 3 failed to encode map e131 with expected crc 2023-12-16T07:50:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[64479]: audit 2023-12-16T07:50:44.335164+0000 mgr.y (mgr.25477) 290 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[64479]: cluster 2023-12-16T07:50:44.499210+0000 mon.a (mon.0) 485 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:50:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[64479]: cluster 2023-12-16T07:50:44.514246+0000 mon.a (mon.0) 486 : cluster 1 osd.5 [v2:172.21.15.169:6808/2407091839,v1:172.21.15.169:6809/2407091839] boot 2023-12-16T07:50:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[64479]: cluster 2023-12-16T07:50:44.514318+0000 mon.a (mon.0) 487 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2023-12-16T07:50:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[64479]: cluster 2023-12-16T07:50:44.518948+0000 osd.6 (osd.6) 46 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[64479]: cluster 2023-12-16T07:50:44.521333+0000 osd.7 (osd.7) 43 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[64479]: cluster 2023-12-16T07:50:44.521769+0000 osd.6 (osd.6) 47 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[64479]: audit 2023-12-16T07:50:44.524653+0000 mon.a (mon.0) 488 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:50:45.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[64479]: cluster 2023-12-16T07:50:44.524772+0000 osd.6 (osd.6) 48 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[64479]: cluster 2023-12-16T07:50:44.527575+0000 osd.6 (osd.6) 49 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[64479]: cluster 2023-12-16T07:50:44.529501+0000 osd.6 (osd.6) 50 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[65864]: cluster 2023-12-16T07:50:44.187238+0000 osd.7 (osd.7) 42 : cluster 3 failed to encode map e131 with expected crc 2023-12-16T07:50:45.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[65864]: audit 2023-12-16T07:50:44.335164+0000 mgr.y (mgr.25477) 290 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:45.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[65864]: cluster 2023-12-16T07:50:44.499210+0000 mon.a (mon.0) 485 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:50:45.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[65864]: cluster 2023-12-16T07:50:44.514246+0000 mon.a (mon.0) 486 : cluster 1 osd.5 [v2:172.21.15.169:6808/2407091839,v1:172.21.15.169:6809/2407091839] boot 2023-12-16T07:50:45.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[65864]: cluster 2023-12-16T07:50:44.514318+0000 mon.a (mon.0) 487 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2023-12-16T07:50:45.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[65864]: cluster 2023-12-16T07:50:44.518948+0000 osd.6 (osd.6) 46 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[65864]: cluster 2023-12-16T07:50:44.521333+0000 osd.7 (osd.7) 43 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[65864]: cluster 2023-12-16T07:50:44.521769+0000 osd.6 (osd.6) 47 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[65864]: audit 2023-12-16T07:50:44.524653+0000 mon.a (mon.0) 488 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:50:45.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[65864]: cluster 2023-12-16T07:50:44.524772+0000 osd.6 (osd.6) 48 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[65864]: cluster 2023-12-16T07:50:44.527575+0000 osd.6 (osd.6) 49 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:45 smithi007 bash[65864]: cluster 2023-12-16T07:50:44.529501+0000 osd.6 (osd.6) 50 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:45 smithi169 bash[51638]: cluster 2023-12-16T07:50:44.187238+0000 osd.7 (osd.7) 42 : cluster 3 failed to encode map e131 with expected crc 2023-12-16T07:50:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:45 smithi169 bash[51638]: audit 2023-12-16T07:50:44.335164+0000 mgr.y (mgr.25477) 290 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:45 smithi169 bash[51638]: cluster 2023-12-16T07:50:44.499210+0000 mon.a (mon.0) 485 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:50:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:45 smithi169 bash[51638]: cluster 2023-12-16T07:50:44.514246+0000 mon.a (mon.0) 486 : cluster 1 osd.5 [v2:172.21.15.169:6808/2407091839,v1:172.21.15.169:6809/2407091839] boot 2023-12-16T07:50:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:45 smithi169 bash[51638]: cluster 2023-12-16T07:50:44.514318+0000 mon.a (mon.0) 487 : cluster 0 osdmap e132: 8 total, 8 up, 8 in 2023-12-16T07:50:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:45 smithi169 bash[51638]: cluster 2023-12-16T07:50:44.518948+0000 osd.6 (osd.6) 46 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:45 smithi169 bash[51638]: cluster 2023-12-16T07:50:44.521333+0000 osd.7 (osd.7) 43 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:45 smithi169 bash[51638]: cluster 2023-12-16T07:50:44.521769+0000 osd.6 (osd.6) 47 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:45 smithi169 bash[51638]: audit 2023-12-16T07:50:44.524653+0000 mon.a (mon.0) 488 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-12-16T07:50:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:45 smithi169 bash[51638]: cluster 2023-12-16T07:50:44.524772+0000 osd.6 (osd.6) 48 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:45 smithi169 bash[51638]: cluster 2023-12-16T07:50:44.527575+0000 osd.6 (osd.6) 49 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:45.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:45 smithi169 bash[51638]: cluster 2023-12-16T07:50:44.529501+0000 osd.6 (osd.6) 50 : cluster 3 failed to encode map e132 with expected crc 2023-12-16T07:50:46.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[64479]: cluster 2023-12-16T07:50:45.256610+0000 mgr.y (mgr.25477) 291 : cluster 0 pgmap v178: 161 pgs: 5 peering, 16 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 64/630 objects degraded (10.159%) 2023-12-16T07:50:46.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[64479]: cluster 2023-12-16T07:50:45.520052+0000 mon.a (mon.0) 489 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2023-12-16T07:50:46.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[64479]: cluster 2023-12-16T07:50:45.520323+0000 osd.6 (osd.6) 51 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[64479]: cluster 2023-12-16T07:50:45.520598+0000 osd.7 (osd.7) 44 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[64479]: cluster 2023-12-16T07:50:45.522691+0000 osd.6 (osd.6) 52 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[64479]: cluster 2023-12-16T07:50:45.527034+0000 osd.7 (osd.7) 45 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[64479]: cluster 2023-12-16T07:50:45.529961+0000 osd.7 (osd.7) 46 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[64479]: cluster 2023-12-16T07:50:45.531777+0000 osd.7 (osd.7) 47 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[65864]: cluster 2023-12-16T07:50:45.256610+0000 mgr.y (mgr.25477) 291 : cluster 0 pgmap v178: 161 pgs: 5 peering, 16 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 64/630 objects degraded (10.159%) 2023-12-16T07:50:46.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[65864]: cluster 2023-12-16T07:50:45.520052+0000 mon.a (mon.0) 489 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2023-12-16T07:50:46.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[65864]: cluster 2023-12-16T07:50:45.520323+0000 osd.6 (osd.6) 51 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[65864]: cluster 2023-12-16T07:50:45.520598+0000 osd.7 (osd.7) 44 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[65864]: cluster 2023-12-16T07:50:45.522691+0000 osd.6 (osd.6) 52 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[65864]: cluster 2023-12-16T07:50:45.527034+0000 osd.7 (osd.7) 45 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[65864]: cluster 2023-12-16T07:50:45.529961+0000 osd.7 (osd.7) 46 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:46 smithi007 bash[65864]: cluster 2023-12-16T07:50:45.531777+0000 osd.7 (osd.7) 47 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:46 smithi169 bash[51638]: cluster 2023-12-16T07:50:45.256610+0000 mgr.y (mgr.25477) 291 : cluster 0 pgmap v178: 161 pgs: 5 peering, 16 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 64/630 objects degraded (10.159%) 2023-12-16T07:50:46.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:46 smithi169 bash[51638]: cluster 2023-12-16T07:50:45.520052+0000 mon.a (mon.0) 489 : cluster 0 osdmap e133: 8 total, 8 up, 8 in 2023-12-16T07:50:46.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:46 smithi169 bash[51638]: cluster 2023-12-16T07:50:45.520323+0000 osd.6 (osd.6) 51 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:46 smithi169 bash[51638]: cluster 2023-12-16T07:50:45.520598+0000 osd.7 (osd.7) 44 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:46 smithi169 bash[51638]: cluster 2023-12-16T07:50:45.522691+0000 osd.6 (osd.6) 52 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:46 smithi169 bash[51638]: cluster 2023-12-16T07:50:45.527034+0000 osd.7 (osd.7) 45 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:46 smithi169 bash[51638]: cluster 2023-12-16T07:50:45.529961+0000 osd.7 (osd.7) 46 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:46.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:46 smithi169 bash[51638]: cluster 2023-12-16T07:50:45.531777+0000 osd.7 (osd.7) 47 : cluster 3 failed to encode map e133 with expected crc 2023-12-16T07:50:48.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:48 smithi007 bash[65864]: cluster 2023-12-16T07:50:47.257915+0000 mgr.y (mgr.25477) 292 : cluster 0 pgmap v180: 161 pgs: 5 peering, 15 active+undersized+degraded, 31 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 54/630 objects degraded (8.571%) 2023-12-16T07:50:48.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:48 smithi007 bash[64479]: cluster 2023-12-16T07:50:47.257915+0000 mgr.y (mgr.25477) 292 : cluster 0 pgmap v180: 161 pgs: 5 peering, 15 active+undersized+degraded, 31 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 54/630 objects degraded (8.571%) 2023-12-16T07:50:48.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:48 smithi169 bash[51638]: cluster 2023-12-16T07:50:47.257915+0000 mgr.y (mgr.25477) 292 : cluster 0 pgmap v180: 161 pgs: 5 peering, 15 active+undersized+degraded, 31 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 54/630 objects degraded (8.571%) 2023-12-16T07:50:49.809 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:49 smithi169 bash[51638]: cluster 2023-12-16T07:50:48.869053+0000 mon.a (mon.0) 490 : cluster 3 Health check update: Degraded data redundancy: 54/630 objects degraded (8.571%), 15 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:49.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:49 smithi007 bash[64479]: cluster 2023-12-16T07:50:48.869053+0000 mon.a (mon.0) 490 : cluster 3 Health check update: Degraded data redundancy: 54/630 objects degraded (8.571%), 15 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:49.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:49 smithi007 bash[65864]: cluster 2023-12-16T07:50:48.869053+0000 mon.a (mon.0) 490 : cluster 3 Health check update: Degraded data redundancy: 54/630 objects degraded (8.571%), 15 pgs degraded (PG_DEGRADED) 2023-12-16T07:50:50.535 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:50:50 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:50:50] "GET /metrics HTTP/1.1" 200 38414 "" "Prometheus/2.33.4" 2023-12-16T07:50:50.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:50 smithi007 bash[65864]: cluster 2023-12-16T07:50:49.259403+0000 mgr.y (mgr.25477) 293 : cluster 0 pgmap v181: 161 pgs: 5 peering, 3 active+undersized+degraded, 2 active+undersized, 151 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 9/630 objects degraded (1.429%) 2023-12-16T07:50:50.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:50 smithi007 bash[64479]: cluster 2023-12-16T07:50:49.259403+0000 mgr.y (mgr.25477) 293 : cluster 0 pgmap v181: 161 pgs: 5 peering, 3 active+undersized+degraded, 2 active+undersized, 151 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 9/630 objects degraded (1.429%) 2023-12-16T07:50:50.937 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:50 smithi169 bash[51638]: cluster 2023-12-16T07:50:49.259403+0000 mgr.y (mgr.25477) 293 : cluster 0 pgmap v181: 161 pgs: 5 peering, 3 active+undersized+degraded, 2 active+undersized, 151 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 9/630 objects degraded (1.429%) 2023-12-16T07:50:51.077 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:50:51.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:51 smithi169 bash[51638]: audit 2023-12-16T07:50:51.058595+0000 mgr.y (mgr.25477) 294 : audit 0 from='client.34491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:51.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:51 smithi169 bash[51638]: cluster 2023-12-16T07:50:51.260584+0000 mgr.y (mgr.25477) 295 : cluster 0 pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:50:51.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:51 smithi169 bash[51638]: audit 2023-12-16T07:50:51.448121+0000 mgr.y (mgr.25477) 296 : audit 0 from='client.34497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:51.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:51 smithi169 bash[51638]: cluster 2023-12-16T07:50:51.528014+0000 mon.a (mon.0) 491 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/630 objects degraded (1.429%), 3 pgs degraded) 2023-12-16T07:50:51.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:51 smithi169 bash[51638]: cluster 2023-12-16T07:50:51.528105+0000 mon.a (mon.0) 492 : cluster 1 Cluster is now healthy 2023-12-16T07:50:51.847 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:50:51.847 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (7m) 68s ago 14m 16.8M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:50:51.847 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (6m) 9s ago 14m 43.8M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:50:51.847 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (5m) 68s ago 12m 47.7M - 3.5 e1d6a67b021e 8d8009d0c83f 2023-12-16T07:50:51.847 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283,8765 running (5m) 9s ago 20m 435M - 19.0.0-191-g6d933094 a29c258b5d6f 5295ba35c2cd 2023-12-16T07:50:51.848 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (6m) 68s ago 22m 493M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:50:51.848 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (5m) 68s ago 22m 52.7M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 5a266b45f3ce 2023-12-16T07:50:51.848 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (4m) 9s ago 20m 43.0M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1fc9bf864dda 2023-12-16T07:50:51.848 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (4m) 68s ago 20m 42.0M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1dc91c31fc27 2023-12-16T07:50:51.848 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (7m) 68s ago 14m 13.0M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:50:51.848 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (6m) 9s ago 14m 14.5M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:50:51.848 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (2m) 68s ago 19m 52.0M 2782M 19.0.0-191-g6d933094 a29c258b5d6f a00724c6d077 2023-12-16T07:50:51.848 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (2m) 68s ago 19m 51.3M 2782M 19.0.0-191-g6d933094 a29c258b5d6f 5e90fce88079 2023-12-16T07:50:51.848 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (104s) 68s ago 18m 46.9M 2782M 19.0.0-191-g6d933094 a29c258b5d6f 906a30291e88 2023-12-16T07:50:51.848 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (72s) 68s ago 18m - 2782M 19.0.0-191-g6d933094 a29c258b5d6f 8e5e7d6ede0f 2023-12-16T07:50:51.848 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (43s) 9s ago 17m 49.9M 3037M 19.0.0-191-g6d933094 a29c258b5d6f b612e0176579 2023-12-16T07:50:51.848 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (13s) 9s ago 16m - 3037M 19.0.0-191-g6d933094 a29c258b5d6f c1ca50cfe550 2023-12-16T07:50:51.848 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (16m) 9s ago 16m 66.1M 3037M 17.2.0 e1d6a67b021e a73bfa272364 2023-12-16T07:50:51.848 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (15m) 9s ago 15m 66.9M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:50:51.849 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (5m) 9s ago 14m 63.8M - 2.33.4 514e6a882f6e 2652ce1b8540 2023-12-16T07:50:51.849 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (12m) 68s ago 12m 88.1M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:50:51.849 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (12m) 9s ago 12m 88.5M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:50:51.849 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (12m) 68s ago 12m 88.0M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:50:51.849 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (12m) 9s ago 12m 88.4M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:50:52.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:51 smithi007 bash[64479]: audit 2023-12-16T07:50:51.058595+0000 mgr.y (mgr.25477) 294 : audit 0 from='client.34491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:52.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:51 smithi007 bash[64479]: cluster 2023-12-16T07:50:51.260584+0000 mgr.y (mgr.25477) 295 : cluster 0 pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:50:52.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:51 smithi007 bash[64479]: audit 2023-12-16T07:50:51.448121+0000 mgr.y (mgr.25477) 296 : audit 0 from='client.34497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:52.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:51 smithi007 bash[64479]: cluster 2023-12-16T07:50:51.528014+0000 mon.a (mon.0) 491 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/630 objects degraded (1.429%), 3 pgs degraded) 2023-12-16T07:50:52.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:51 smithi007 bash[64479]: cluster 2023-12-16T07:50:51.528105+0000 mon.a (mon.0) 492 : cluster 1 Cluster is now healthy 2023-12-16T07:50:52.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:51 smithi007 bash[65864]: audit 2023-12-16T07:50:51.058595+0000 mgr.y (mgr.25477) 294 : audit 0 from='client.34491 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:52.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:51 smithi007 bash[65864]: cluster 2023-12-16T07:50:51.260584+0000 mgr.y (mgr.25477) 295 : cluster 0 pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:50:52.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:51 smithi007 bash[65864]: audit 2023-12-16T07:50:51.448121+0000 mgr.y (mgr.25477) 296 : audit 0 from='client.34497 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:52.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:51 smithi007 bash[65864]: cluster 2023-12-16T07:50:51.528014+0000 mon.a (mon.0) 491 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/630 objects degraded (1.429%), 3 pgs degraded) 2023-12-16T07:50:52.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:51 smithi007 bash[65864]: cluster 2023-12-16T07:50:51.528105+0000 mon.a (mon.0) 492 : cluster 1 Cluster is now healthy 2023-12-16T07:50:52.305 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:50:52.305 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:50:52.305 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 3 2023-12-16T07:50:52.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:50:52.305 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:50:52.305 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:50:52.305 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:50:52.305 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:50:52.305 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2023-12-16T07:50:52.305 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 6 2023-12-16T07:50:52.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:50:52.306 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:50:52.306 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:50:52.306 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:50:52.306 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:50:52.306 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2023-12-16T07:50:52.306 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 11 2023-12-16T07:50:52.306 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:50:52.306 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:50:52.689 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:50:52.689 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:50:52.689 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:50:52.689 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:50:52.689 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2023-12-16T07:50:52.689 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr", 2023-12-16T07:50:52.689 INFO:teuthology.orchestra.run.smithi007.stdout: "mon" 2023-12-16T07:50:52.689 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:50:52.690 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "11/23 daemons upgraded", 2023-12-16T07:50:52.690 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading osd daemons", 2023-12-16T07:50:52.690 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:50:52.690 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:50:52.911 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:52 smithi169 bash[51638]: audit 2023-12-16T07:50:51.820487+0000 mgr.y (mgr.25477) 297 : audit 0 from='client.34503 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:52.911 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:52 smithi169 bash[51638]: audit 2023-12-16T07:50:52.302231+0000 mon.c (mon.1) 14 : audit 0 from='client.? 172.21.15.7:0/479905885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:53.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:52 smithi007 bash[65864]: audit 2023-12-16T07:50:51.820487+0000 mgr.y (mgr.25477) 297 : audit 0 from='client.34503 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:53.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:52 smithi007 bash[65864]: audit 2023-12-16T07:50:52.302231+0000 mon.c (mon.1) 14 : audit 0 from='client.? 172.21.15.7:0/479905885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:53.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:52 smithi007 bash[64479]: audit 2023-12-16T07:50:51.820487+0000 mgr.y (mgr.25477) 297 : audit 0 from='client.34503 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:53.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:52 smithi007 bash[64479]: audit 2023-12-16T07:50:52.302231+0000 mon.c (mon.1) 14 : audit 0 from='client.? 172.21.15.7:0/479905885' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:53.142 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:50:53.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:53 smithi169 bash[51638]: audit 2023-12-16T07:50:52.685582+0000 mgr.y (mgr.25477) 298 : audit 0 from='client.44443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:53.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:53 smithi169 bash[51638]: audit 2023-12-16T07:50:53.139926+0000 mon.a (mon.0) 493 : audit 0 from='client.? 172.21.15.7:0/1667443251' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:50:53.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:53 smithi169 bash[51638]: audit 2023-12-16T07:50:53.212627+0000 mon.a (mon.0) 494 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:53.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:53 smithi169 bash[51638]: audit 2023-12-16T07:50:53.214801+0000 mon.a (mon.0) 495 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:50:53.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:53 smithi169 bash[51638]: cluster 2023-12-16T07:50:53.261483+0000 mgr.y (mgr.25477) 299 : cluster 0 pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-12-16T07:50:53.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:53 smithi169 bash[51638]: audit 2023-12-16T07:50:53.495663+0000 mon.a (mon.0) 496 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:53.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:53 smithi169 bash[51638]: audit 2023-12-16T07:50:53.507442+0000 mon.a (mon.0) 497 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:53.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:53 smithi169 bash[51638]: audit 2023-12-16T07:50:53.509472+0000 mon.a (mon.0) 498 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:50:53.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:53 smithi169 bash[51638]: audit 2023-12-16T07:50:53.511467+0000 mon.a (mon.0) 499 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:50:53.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:53 smithi169 bash[51638]: audit 2023-12-16T07:50:53.519785+0000 mon.a (mon.0) 500 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[64479]: audit 2023-12-16T07:50:52.685582+0000 mgr.y (mgr.25477) 298 : audit 0 from='client.44443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[64479]: audit 2023-12-16T07:50:53.139926+0000 mon.a (mon.0) 493 : audit 0 from='client.? 172.21.15.7:0/1667443251' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:50:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[64479]: audit 2023-12-16T07:50:53.212627+0000 mon.a (mon.0) 494 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[64479]: audit 2023-12-16T07:50:53.214801+0000 mon.a (mon.0) 495 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:50:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[64479]: cluster 2023-12-16T07:50:53.261483+0000 mgr.y (mgr.25477) 299 : cluster 0 pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-12-16T07:50:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[64479]: audit 2023-12-16T07:50:53.495663+0000 mon.a (mon.0) 496 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[64479]: audit 2023-12-16T07:50:53.507442+0000 mon.a (mon.0) 497 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[64479]: audit 2023-12-16T07:50:53.509472+0000 mon.a (mon.0) 498 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:50:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[64479]: audit 2023-12-16T07:50:53.511467+0000 mon.a (mon.0) 499 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:50:54.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[64479]: audit 2023-12-16T07:50:53.519785+0000 mon.a (mon.0) 500 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:54.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[65864]: audit 2023-12-16T07:50:52.685582+0000 mgr.y (mgr.25477) 298 : audit 0 from='client.44443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:50:54.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[65864]: audit 2023-12-16T07:50:53.139926+0000 mon.a (mon.0) 493 : audit 0 from='client.? 172.21.15.7:0/1667443251' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:50:54.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[65864]: audit 2023-12-16T07:50:53.212627+0000 mon.a (mon.0) 494 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:54.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[65864]: audit 2023-12-16T07:50:53.214801+0000 mon.a (mon.0) 495 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:50:54.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[65864]: cluster 2023-12-16T07:50:53.261483+0000 mgr.y (mgr.25477) 299 : cluster 0 pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-12-16T07:50:54.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[65864]: audit 2023-12-16T07:50:53.495663+0000 mon.a (mon.0) 496 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:54.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[65864]: audit 2023-12-16T07:50:53.507442+0000 mon.a (mon.0) 497 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:54.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[65864]: audit 2023-12-16T07:50:53.509472+0000 mon.a (mon.0) 498 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:50:54.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[65864]: audit 2023-12-16T07:50:53.511467+0000 mon.a (mon.0) 499 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:50:54.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:53 smithi007 bash[65864]: audit 2023-12-16T07:50:53.519785+0000 mon.a (mon.0) 500 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:54.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[64479]: audit 2023-12-16T07:50:53.605952+0000 mon.a (mon.0) 501 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:50:54.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[64479]: audit 2023-12-16T07:50:53.609182+0000 mon.a (mon.0) 502 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:54.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[64479]: audit 2023-12-16T07:50:53.611693+0000 mon.a (mon.0) 503 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:54.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[64479]: audit 2023-12-16T07:50:53.614427+0000 mon.a (mon.0) 504 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:54.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[64479]: audit 2023-12-16T07:50:53.618139+0000 mon.a (mon.0) 505 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-12-16T07:50:54.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[64479]: audit 2023-12-16T07:50:53.618495+0000 mgr.y (mgr.25477) 300 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-12-16T07:50:54.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[64479]: cephadm 2023-12-16T07:50:53.620889+0000 mgr.y (mgr.25477) 301 : cephadm 1 Upgrade: osd.6 is safe to restart 2023-12-16T07:50:54.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[64479]: audit 2023-12-16T07:50:54.346434+0000 mgr.y (mgr.25477) 302 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:54.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[65864]: audit 2023-12-16T07:50:53.605952+0000 mon.a (mon.0) 501 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:50:54.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[65864]: audit 2023-12-16T07:50:53.609182+0000 mon.a (mon.0) 502 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:54.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[65864]: audit 2023-12-16T07:50:53.611693+0000 mon.a (mon.0) 503 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:54.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[65864]: audit 2023-12-16T07:50:53.614427+0000 mon.a (mon.0) 504 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:54.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[65864]: audit 2023-12-16T07:50:53.618139+0000 mon.a (mon.0) 505 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-12-16T07:50:54.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[65864]: audit 2023-12-16T07:50:53.618495+0000 mgr.y (mgr.25477) 300 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-12-16T07:50:54.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[65864]: cephadm 2023-12-16T07:50:53.620889+0000 mgr.y (mgr.25477) 301 : cephadm 1 Upgrade: osd.6 is safe to restart 2023-12-16T07:50:54.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:54 smithi007 bash[65864]: audit 2023-12-16T07:50:54.346434+0000 mgr.y (mgr.25477) 302 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:54.916 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:54 smithi169 bash[51638]: audit 2023-12-16T07:50:53.605952+0000 mon.a (mon.0) 501 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:50:54.916 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:54 smithi169 bash[51638]: audit 2023-12-16T07:50:53.609182+0000 mon.a (mon.0) 502 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:54.916 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:54 smithi169 bash[51638]: audit 2023-12-16T07:50:53.611693+0000 mon.a (mon.0) 503 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:54.916 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:54 smithi169 bash[51638]: audit 2023-12-16T07:50:53.614427+0000 mon.a (mon.0) 504 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:50:54.916 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:54 smithi169 bash[51638]: audit 2023-12-16T07:50:53.618139+0000 mon.a (mon.0) 505 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-12-16T07:50:54.916 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:54 smithi169 bash[51638]: audit 2023-12-16T07:50:53.618495+0000 mgr.y (mgr.25477) 300 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2023-12-16T07:50:54.917 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:54 smithi169 bash[51638]: cephadm 2023-12-16T07:50:53.620889+0000 mgr.y (mgr.25477) 301 : cephadm 1 Upgrade: osd.6 is safe to restart 2023-12-16T07:50:54.917 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:54 smithi169 bash[51638]: audit 2023-12-16T07:50:54.346434+0000 mgr.y (mgr.25477) 302 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:50:55.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:55 smithi169 bash[51638]: cluster 2023-12-16T07:50:55.263057+0000 mgr.y (mgr.25477) 303 : cluster 0 pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:50:56.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:55 smithi007 bash[64479]: cluster 2023-12-16T07:50:55.263057+0000 mgr.y (mgr.25477) 303 : cluster 0 pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:50:56.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:55 smithi007 bash[65864]: cluster 2023-12-16T07:50:55.263057+0000 mgr.y (mgr.25477) 303 : cluster 0 pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:50:57.156 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:56 smithi169 bash[51638]: cephadm 2023-12-16T07:50:55.892035+0000 mgr.y (mgr.25477) 304 : cephadm 1 Upgrade: Updating osd.6 2023-12-16T07:50:57.156 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:56 smithi169 bash[51638]: audit 2023-12-16T07:50:55.899355+0000 mon.a (mon.0) 506 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:57.157 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:56 smithi169 bash[51638]: audit 2023-12-16T07:50:55.904722+0000 mon.a (mon.0) 507 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-12-16T07:50:57.157 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:56 smithi169 bash[51638]: audit 2023-12-16T07:50:55.906300+0000 mon.a (mon.0) 508 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:50:57.157 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:56 smithi169 bash[51638]: cephadm 2023-12-16T07:50:55.910101+0000 mgr.y (mgr.25477) 305 : cephadm 1 Deploying daemon osd.6 on smithi169 2023-12-16T07:50:57.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:56 smithi007 bash[64479]: cephadm 2023-12-16T07:50:55.892035+0000 mgr.y (mgr.25477) 304 : cephadm 1 Upgrade: Updating osd.6 2023-12-16T07:50:57.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:56 smithi007 bash[64479]: audit 2023-12-16T07:50:55.899355+0000 mon.a (mon.0) 506 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:57.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:56 smithi007 bash[64479]: audit 2023-12-16T07:50:55.904722+0000 mon.a (mon.0) 507 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-12-16T07:50:57.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:56 smithi007 bash[64479]: audit 2023-12-16T07:50:55.906300+0000 mon.a (mon.0) 508 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:50:57.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:56 smithi007 bash[64479]: cephadm 2023-12-16T07:50:55.910101+0000 mgr.y (mgr.25477) 305 : cephadm 1 Deploying daemon osd.6 on smithi169 2023-12-16T07:50:57.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:56 smithi007 bash[65864]: cephadm 2023-12-16T07:50:55.892035+0000 mgr.y (mgr.25477) 304 : cephadm 1 Upgrade: Updating osd.6 2023-12-16T07:50:57.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:56 smithi007 bash[65864]: audit 2023-12-16T07:50:55.899355+0000 mon.a (mon.0) 506 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:57.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:56 smithi007 bash[65864]: audit 2023-12-16T07:50:55.904722+0000 mon.a (mon.0) 507 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-12-16T07:50:57.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:56 smithi007 bash[65864]: audit 2023-12-16T07:50:55.906300+0000 mon.a (mon.0) 508 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:50:57.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:56 smithi007 bash[65864]: cephadm 2023-12-16T07:50:55.910101+0000 mgr.y (mgr.25477) 305 : cephadm 1 Deploying daemon osd.6 on smithi169 2023-12-16T07:50:58.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:57 smithi169 bash[51638]: cluster 2023-12-16T07:50:57.263951+0000 mgr.y (mgr.25477) 306 : cluster 0 pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2023-12-16T07:50:58.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:57 smithi007 bash[64479]: cluster 2023-12-16T07:50:57.263951+0000 mgr.y (mgr.25477) 306 : cluster 0 pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2023-12-16T07:50:58.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:57 smithi007 bash[65864]: cluster 2023-12-16T07:50:57.263951+0000 mgr.y (mgr.25477) 306 : cluster 0 pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2023-12-16T07:50:59.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:50:59 smithi169 bash[51638]: audit 2023-12-16T07:50:58.492126+0000 mon.a (mon.0) 509 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:59.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:50:59 smithi007 bash[64479]: audit 2023-12-16T07:50:58.492126+0000 mon.a (mon.0) 509 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:50:59.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:50:59 smithi007 bash[65864]: audit 2023-12-16T07:50:58.492126+0000 mon.a (mon.0) 509 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:00.495 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:00 smithi169 systemd[1]: Stopping Ceph osd.6 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:51:00.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:00 smithi007 bash[64479]: cluster 2023-12-16T07:50:59.265248+0000 mgr.y (mgr.25477) 307 : cluster 0 pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:51:00.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:51:00 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:51:00] "GET /metrics HTTP/1.1" 200 38423 "" "Prometheus/2.33.4" 2023-12-16T07:51:00.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:00 smithi007 bash[65864]: cluster 2023-12-16T07:50:59.265248+0000 mgr.y (mgr.25477) 307 : cluster 0 pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:51:00.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:00 smithi169 bash[51638]: cluster 2023-12-16T07:50:59.265248+0000 mgr.y (mgr.25477) 307 : cluster 0 pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:51:00.933 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:00 smithi169 bash[34375]: debug 2023-12-16T07:51:00.582+0000 7f83463a9700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:51:00.933 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:00 smithi169 bash[34375]: debug 2023-12-16T07:51:00.582+0000 7f83463a9700 -1 osd.6 133 *** Got signal Terminated *** 2023-12-16T07:51:00.933 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:00 smithi169 bash[34375]: debug 2023-12-16T07:51:00.582+0000 7f83463a9700 -1 osd.6 133 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:51:01.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:01 smithi007 bash[64479]: cluster 2023-12-16T07:51:00.587290+0000 mon.a (mon.0) 510 : cluster 1 osd.6 marked itself down and dead 2023-12-16T07:51:01.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:01 smithi007 bash[65864]: cluster 2023-12-16T07:51:00.587290+0000 mon.a (mon.0) 510 : cluster 1 osd.6 marked itself down and dead 2023-12-16T07:51:01.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:01 smithi169 bash[51638]: cluster 2023-12-16T07:51:00.587290+0000 mon.a (mon.0) 510 : cluster 1 osd.6 marked itself down and dead 2023-12-16T07:51:02.798 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:02 smithi169 bash[57569]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-6 2023-12-16T07:51:02.798 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:02 smithi169 bash[51638]: cluster 2023-12-16T07:51:01.266458+0000 mgr.y (mgr.25477) 308 : cluster 0 pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:51:02.798 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:02 smithi169 bash[51638]: cluster 2023-12-16T07:51:01.493734+0000 mon.a (mon.0) 511 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:51:02.798 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:02 smithi169 bash[51638]: cluster 2023-12-16T07:51:01.510230+0000 mon.a (mon.0) 512 : cluster 0 osdmap e134: 8 total, 7 up, 8 in 2023-12-16T07:51:02.798 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:02 smithi169 bash[51638]: cluster 2023-12-16T07:51:01.517592+0000 osd.7 (osd.7) 48 : cluster 3 failed to encode map e134 with expected crc 2023-12-16T07:51:02.799 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:02 smithi169 bash[51638]: cluster 2023-12-16T07:51:01.520018+0000 osd.7 (osd.7) 49 : cluster 3 failed to encode map e134 with expected crc 2023-12-16T07:51:02.799 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:02 smithi169 bash[51638]: cluster 2023-12-16T07:51:01.522531+0000 osd.7 (osd.7) 50 : cluster 3 failed to encode map e134 with expected crc 2023-12-16T07:51:02.799 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:02 smithi169 bash[51638]: cluster 2023-12-16T07:51:01.524689+0000 osd.7 (osd.7) 51 : cluster 3 failed to encode map e134 with expected crc 2023-12-16T07:51:02.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:02 smithi007 bash[64479]: cluster 2023-12-16T07:51:01.266458+0000 mgr.y (mgr.25477) 308 : cluster 0 pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:51:02.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:02 smithi007 bash[64479]: cluster 2023-12-16T07:51:01.493734+0000 mon.a (mon.0) 511 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:51:02.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:02 smithi007 bash[64479]: cluster 2023-12-16T07:51:01.510230+0000 mon.a (mon.0) 512 : cluster 0 osdmap e134: 8 total, 7 up, 8 in 2023-12-16T07:51:02.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:02 smithi007 bash[64479]: cluster 2023-12-16T07:51:01.517592+0000 osd.7 (osd.7) 48 : cluster 3 failed to encode map e134 with expected crc 2023-12-16T07:51:02.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:02 smithi007 bash[64479]: cluster 2023-12-16T07:51:01.520018+0000 osd.7 (osd.7) 49 : cluster 3 failed to encode map e134 with expected crc 2023-12-16T07:51:02.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:02 smithi007 bash[64479]: cluster 2023-12-16T07:51:01.522531+0000 osd.7 (osd.7) 50 : cluster 3 failed to encode map e134 with expected crc 2023-12-16T07:51:02.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:02 smithi007 bash[64479]: cluster 2023-12-16T07:51:01.524689+0000 osd.7 (osd.7) 51 : cluster 3 failed to encode map e134 with expected crc 2023-12-16T07:51:02.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:02 smithi007 bash[65864]: cluster 2023-12-16T07:51:01.266458+0000 mgr.y (mgr.25477) 308 : cluster 0 pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:51:02.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:02 smithi007 bash[65864]: cluster 2023-12-16T07:51:01.493734+0000 mon.a (mon.0) 511 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:51:02.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:02 smithi007 bash[65864]: cluster 2023-12-16T07:51:01.510230+0000 mon.a (mon.0) 512 : cluster 0 osdmap e134: 8 total, 7 up, 8 in 2023-12-16T07:51:02.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:02 smithi007 bash[65864]: cluster 2023-12-16T07:51:01.517592+0000 osd.7 (osd.7) 48 : cluster 3 failed to encode map e134 with expected crc 2023-12-16T07:51:02.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:02 smithi007 bash[65864]: cluster 2023-12-16T07:51:01.520018+0000 osd.7 (osd.7) 49 : cluster 3 failed to encode map e134 with expected crc 2023-12-16T07:51:02.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:02 smithi007 bash[65864]: cluster 2023-12-16T07:51:01.522531+0000 osd.7 (osd.7) 50 : cluster 3 failed to encode map e134 with expected crc 2023-12-16T07:51:02.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:02 smithi007 bash[65864]: cluster 2023-12-16T07:51:01.524689+0000 osd.7 (osd.7) 51 : cluster 3 failed to encode map e134 with expected crc 2023-12-16T07:51:03.183 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:02 smithi169 bash[57643]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-6 2023-12-16T07:51:03.770 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:03 smithi169 bash[51638]: cluster 2023-12-16T07:51:02.510597+0000 mon.a (mon.0) 513 : cluster 0 osdmap e135: 8 total, 7 up, 8 in 2023-12-16T07:51:03.770 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:03 smithi169 bash[51638]: cluster 2023-12-16T07:51:02.516556+0000 osd.7 (osd.7) 52 : cluster 3 failed to encode map e135 with expected crc 2023-12-16T07:51:03.770 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:03 smithi169 bash[51638]: cluster 2023-12-16T07:51:02.522201+0000 osd.7 (osd.7) 53 : cluster 3 failed to encode map e135 with expected crc 2023-12-16T07:51:03.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:03 smithi007 bash[64479]: cluster 2023-12-16T07:51:02.510597+0000 mon.a (mon.0) 513 : cluster 0 osdmap e135: 8 total, 7 up, 8 in 2023-12-16T07:51:03.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:03 smithi007 bash[64479]: cluster 2023-12-16T07:51:02.516556+0000 osd.7 (osd.7) 52 : cluster 3 failed to encode map e135 with expected crc 2023-12-16T07:51:03.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:03 smithi007 bash[64479]: cluster 2023-12-16T07:51:02.522201+0000 osd.7 (osd.7) 53 : cluster 3 failed to encode map e135 with expected crc 2023-12-16T07:51:03.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:03 smithi007 bash[65864]: cluster 2023-12-16T07:51:02.510597+0000 mon.a (mon.0) 513 : cluster 0 osdmap e135: 8 total, 7 up, 8 in 2023-12-16T07:51:03.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:03 smithi007 bash[65864]: cluster 2023-12-16T07:51:02.516556+0000 osd.7 (osd.7) 52 : cluster 3 failed to encode map e135 with expected crc 2023-12-16T07:51:03.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:03 smithi007 bash[65864]: cluster 2023-12-16T07:51:02.522201+0000 osd.7 (osd.7) 53 : cluster 3 failed to encode map e135 with expected crc 2023-12-16T07:51:04.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:04 smithi169 bash[51638]: cluster 2023-12-16T07:51:03.267535+0000 mgr.y (mgr.25477) 309 : cluster 0 pgmap v190: 161 pgs: 10 active+undersized, 5 active+undersized+degraded, 7 stale+active+clean, 139 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 24/630 objects degraded (3.810%) 2023-12-16T07:51:04.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:04 smithi169 bash[51638]: cluster 2023-12-16T07:51:03.506747+0000 mon.a (mon.0) 514 : cluster 3 Health check failed: Degraded data redundancy: 24/630 objects degraded (3.810%), 5 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:04.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:04 smithi007 bash[64479]: cluster 2023-12-16T07:51:03.267535+0000 mgr.y (mgr.25477) 309 : cluster 0 pgmap v190: 161 pgs: 10 active+undersized, 5 active+undersized+degraded, 7 stale+active+clean, 139 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 24/630 objects degraded (3.810%) 2023-12-16T07:51:04.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:04 smithi007 bash[64479]: cluster 2023-12-16T07:51:03.506747+0000 mon.a (mon.0) 514 : cluster 3 Health check failed: Degraded data redundancy: 24/630 objects degraded (3.810%), 5 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:04.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:04 smithi007 bash[65864]: cluster 2023-12-16T07:51:03.267535+0000 mgr.y (mgr.25477) 309 : cluster 0 pgmap v190: 161 pgs: 10 active+undersized, 5 active+undersized+degraded, 7 stale+active+clean, 139 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 24/630 objects degraded (3.810%) 2023-12-16T07:51:04.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:04 smithi007 bash[65864]: cluster 2023-12-16T07:51:03.506747+0000 mon.a (mon.0) 514 : cluster 3 Health check failed: Degraded data redundancy: 24/630 objects degraded (3.810%), 5 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:05.064 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:04 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.6.service: Succeeded. 2023-12-16T07:51:05.064 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:04 smithi169 systemd[1]: Stopped Ceph osd.6 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:51:05.433 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:05 smithi169 systemd[1]: Started Ceph osd.6 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:51:05.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:05 smithi007 bash[65864]: audit 2023-12-16T07:51:04.354684+0000 mgr.y (mgr.25477) 310 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:05.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:05 smithi007 bash[65864]: audit 2023-12-16T07:51:05.115246+0000 mon.a (mon.0) 515 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:05.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:05 smithi007 bash[65864]: audit 2023-12-16T07:51:05.124221+0000 mon.a (mon.0) 516 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:05.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:05 smithi007 bash[64479]: audit 2023-12-16T07:51:04.354684+0000 mgr.y (mgr.25477) 310 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:05.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:05 smithi007 bash[64479]: audit 2023-12-16T07:51:05.115246+0000 mon.a (mon.0) 515 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:05.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:05 smithi007 bash[64479]: audit 2023-12-16T07:51:05.124221+0000 mon.a (mon.0) 516 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:05.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:05 smithi169 bash[51638]: audit 2023-12-16T07:51:04.354684+0000 mgr.y (mgr.25477) 310 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:05.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:05 smithi169 bash[51638]: audit 2023-12-16T07:51:05.115246+0000 mon.a (mon.0) 515 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:05.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:05 smithi169 bash[51638]: audit 2023-12-16T07:51:05.124221+0000 mon.a (mon.0) 516 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:06.817 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:06 smithi169 bash[51638]: cluster 2023-12-16T07:51:05.269244+0000 mgr.y (mgr.25477) 311 : cluster 0 pgmap v191: 161 pgs: 31 active+undersized, 18 active+undersized+degraded, 112 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 66/630 objects degraded (10.476%) 2023-12-16T07:51:06.818 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:06 smithi169 bash[57814]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-16T07:51:06.818 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:06 smithi169 bash[57814]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2023-12-16T07:51:06.818 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:06 smithi169 bash[57814]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2023-12-16T07:51:06.818 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:06 smithi169 bash[57814]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-12-16T07:51:06.818 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:06 smithi169 bash[57814]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2023-12-16T07:51:06.818 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:06 smithi169 bash[57814]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-16T07:51:06.818 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:06 smithi169 bash[57814]: --> ceph-volume raw activate successful for osd ID: 6 2023-12-16T07:51:06.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:06 smithi007 bash[65864]: cluster 2023-12-16T07:51:05.269244+0000 mgr.y (mgr.25477) 311 : cluster 0 pgmap v191: 161 pgs: 31 active+undersized, 18 active+undersized+degraded, 112 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 66/630 objects degraded (10.476%) 2023-12-16T07:51:06.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:06 smithi007 bash[64479]: cluster 2023-12-16T07:51:05.269244+0000 mgr.y (mgr.25477) 311 : cluster 0 pgmap v191: 161 pgs: 31 active+undersized, 18 active+undersized+degraded, 112 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 66/630 objects degraded (10.476%) 2023-12-16T07:51:08.433 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:08 smithi169 bash[58112]: debug 2023-12-16T07:51:08.194+0000 7f9299019700 1 -- 172.21.15.169:0/4179002111 <== mon.2 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55680aa5f6c0 con 0x55680aae4800 2023-12-16T07:51:08.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:08 smithi007 bash[65864]: cluster 2023-12-16T07:51:07.270479+0000 mgr.y (mgr.25477) 312 : cluster 0 pgmap v192: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2023-12-16T07:51:08.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:08 smithi007 bash[65864]: audit 2023-12-16T07:51:08.213202+0000 mon.a (mon.0) 517 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:08.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:08 smithi007 bash[65864]: audit 2023-12-16T07:51:08.215564+0000 mon.a (mon.0) 518 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:51:08.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:08 smithi007 bash[64479]: cluster 2023-12-16T07:51:07.270479+0000 mgr.y (mgr.25477) 312 : cluster 0 pgmap v192: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2023-12-16T07:51:08.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:08 smithi007 bash[64479]: audit 2023-12-16T07:51:08.213202+0000 mon.a (mon.0) 517 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:08.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:08 smithi007 bash[64479]: audit 2023-12-16T07:51:08.215564+0000 mon.a (mon.0) 518 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:51:08.933 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:08 smithi169 bash[58112]: debug 2023-12-16T07:51:08.746+0000 7f92a219f640 -1 Falling back to public interface 2023-12-16T07:51:08.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:08 smithi169 bash[51638]: cluster 2023-12-16T07:51:07.270479+0000 mgr.y (mgr.25477) 312 : cluster 0 pgmap v192: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2023-12-16T07:51:08.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:08 smithi169 bash[51638]: audit 2023-12-16T07:51:08.213202+0000 mon.a (mon.0) 517 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:08.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:08 smithi169 bash[51638]: audit 2023-12-16T07:51:08.215564+0000 mon.a (mon.0) 518 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:51:09.817 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:09 smithi007 bash[64479]: cluster 2023-12-16T07:51:08.875692+0000 mon.a (mon.0) 519 : cluster 3 Health check update: Degraded data redundancy: 76/630 objects degraded (12.063%), 20 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:09.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:09 smithi007 bash[65864]: cluster 2023-12-16T07:51:08.875692+0000 mon.a (mon.0) 519 : cluster 3 Health check update: Degraded data redundancy: 76/630 objects degraded (12.063%), 20 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:09.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:09 smithi169 bash[51638]: cluster 2023-12-16T07:51:08.875692+0000 mon.a (mon.0) 519 : cluster 3 Health check update: Degraded data redundancy: 76/630 objects degraded (12.063%), 20 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:10.533 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:51:10 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:51:10] "GET /metrics HTTP/1.1" 200 38427 "" "Prometheus/2.33.4" 2023-12-16T07:51:10.533 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:10 smithi007 bash[64479]: cluster 2023-12-16T07:51:09.271970+0000 mgr.y (mgr.25477) 313 : cluster 0 pgmap v193: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2023-12-16T07:51:10.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:10 smithi007 bash[65864]: cluster 2023-12-16T07:51:09.271970+0000 mgr.y (mgr.25477) 313 : cluster 0 pgmap v193: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2023-12-16T07:51:10.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:10 smithi169 bash[51638]: cluster 2023-12-16T07:51:09.271970+0000 mgr.y (mgr.25477) 313 : cluster 0 pgmap v193: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2023-12-16T07:51:12.183 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:11 smithi169 bash[58112]: debug 2023-12-16T07:51:11.826+0000 7f92a219f640 -1 osd.6 0 read_superblock omap replica is missing. 2023-12-16T07:51:12.183 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:11 smithi169 bash[58112]: debug 2023-12-16T07:51:11.862+0000 7f92a219f640 -1 osd.6 133 log_to_monitors true 2023-12-16T07:51:12.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:12 smithi007 bash[64479]: cluster 2023-12-16T07:51:11.273566+0000 mgr.y (mgr.25477) 314 : cluster 0 pgmap v194: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2023-12-16T07:51:12.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:12 smithi007 bash[64479]: audit 2023-12-16T07:51:11.548269+0000 mon.a (mon.0) 520 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:12.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:12 smithi007 bash[64479]: audit 2023-12-16T07:51:11.558299+0000 mon.a (mon.0) 521 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:12.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:12 smithi007 bash[64479]: audit 2023-12-16T07:51:11.868215+0000 mon.a (mon.0) 522 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-12-16T07:51:12.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:12 smithi007 bash[64479]: audit 2023-12-16T07:51:11.869641+0000 mon.b (mon.2) 7 : audit 1 from='osd.6 [v2:172.21.15.169:6816/2677142792,v1:172.21.15.169:6817/2677142792]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-12-16T07:51:12.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:12 smithi007 bash[65864]: cluster 2023-12-16T07:51:11.273566+0000 mgr.y (mgr.25477) 314 : cluster 0 pgmap v194: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2023-12-16T07:51:12.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:12 smithi007 bash[65864]: audit 2023-12-16T07:51:11.548269+0000 mon.a (mon.0) 520 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:12.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:12 smithi007 bash[65864]: audit 2023-12-16T07:51:11.558299+0000 mon.a (mon.0) 521 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:12.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:12 smithi007 bash[65864]: audit 2023-12-16T07:51:11.868215+0000 mon.a (mon.0) 522 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-12-16T07:51:12.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:12 smithi007 bash[65864]: audit 2023-12-16T07:51:11.869641+0000 mon.b (mon.2) 7 : audit 1 from='osd.6 [v2:172.21.15.169:6816/2677142792,v1:172.21.15.169:6817/2677142792]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-12-16T07:51:12.933 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:51:12 smithi169 bash[58112]: debug 2023-12-16T07:51:12.606+0000 7f9291e0d700 -1 osd.6 133 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:51:12.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:12 smithi169 bash[51638]: cluster 2023-12-16T07:51:11.273566+0000 mgr.y (mgr.25477) 314 : cluster 0 pgmap v194: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2023-12-16T07:51:12.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:12 smithi169 bash[51638]: audit 2023-12-16T07:51:11.548269+0000 mon.a (mon.0) 520 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:12.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:12 smithi169 bash[51638]: audit 2023-12-16T07:51:11.558299+0000 mon.a (mon.0) 521 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:12.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:12 smithi169 bash[51638]: audit 2023-12-16T07:51:11.868215+0000 mon.a (mon.0) 522 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-12-16T07:51:12.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:12 smithi169 bash[51638]: audit 2023-12-16T07:51:11.869641+0000 mon.b (mon.2) 7 : audit 1 from='osd.6 [v2:172.21.15.169:6816/2677142792,v1:172.21.15.169:6817/2677142792]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-12-16T07:51:13.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[64479]: audit 2023-12-16T07:51:12.557629+0000 mon.a (mon.0) 523 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-12-16T07:51:13.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[64479]: cluster 2023-12-16T07:51:12.561830+0000 osd.7 (osd.7) 54 : cluster 3 failed to encode map e136 with expected crc 2023-12-16T07:51:13.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[64479]: cluster 2023-12-16T07:51:12.562068+0000 mon.a (mon.0) 524 : cluster 0 osdmap e136: 8 total, 7 up, 8 in 2023-12-16T07:51:13.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[64479]: audit 2023-12-16T07:51:12.563881+0000 mon.a (mon.0) 525 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:51:13.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[64479]: audit 2023-12-16T07:51:12.565588+0000 mon.b (mon.2) 8 : audit 1 from='osd.6 [v2:172.21.15.169:6816/2677142792,v1:172.21.15.169:6817/2677142792]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:51:13.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[64479]: audit 2023-12-16T07:51:12.604683+0000 mon.a (mon.0) 526 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:13.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[64479]: audit 2023-12-16T07:51:12.615454+0000 mon.a (mon.0) 527 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:13.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[64479]: audit 2023-12-16T07:51:13.526286+0000 mon.a (mon.0) 528 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:13.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[65864]: audit 2023-12-16T07:51:12.557629+0000 mon.a (mon.0) 523 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-12-16T07:51:13.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[65864]: cluster 2023-12-16T07:51:12.561830+0000 osd.7 (osd.7) 54 : cluster 3 failed to encode map e136 with expected crc 2023-12-16T07:51:13.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[65864]: cluster 2023-12-16T07:51:12.562068+0000 mon.a (mon.0) 524 : cluster 0 osdmap e136: 8 total, 7 up, 8 in 2023-12-16T07:51:13.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[65864]: audit 2023-12-16T07:51:12.563881+0000 mon.a (mon.0) 525 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:51:13.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[65864]: audit 2023-12-16T07:51:12.565588+0000 mon.b (mon.2) 8 : audit 1 from='osd.6 [v2:172.21.15.169:6816/2677142792,v1:172.21.15.169:6817/2677142792]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:51:13.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[65864]: audit 2023-12-16T07:51:12.604683+0000 mon.a (mon.0) 526 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:13.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[65864]: audit 2023-12-16T07:51:12.615454+0000 mon.a (mon.0) 527 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:13.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:13 smithi007 bash[65864]: audit 2023-12-16T07:51:13.526286+0000 mon.a (mon.0) 528 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:13.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:13 smithi169 bash[51638]: audit 2023-12-16T07:51:12.557629+0000 mon.a (mon.0) 523 : audit 1 from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-12-16T07:51:13.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:13 smithi169 bash[51638]: cluster 2023-12-16T07:51:12.561830+0000 osd.7 (osd.7) 54 : cluster 3 failed to encode map e136 with expected crc 2023-12-16T07:51:13.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:13 smithi169 bash[51638]: cluster 2023-12-16T07:51:12.562068+0000 mon.a (mon.0) 524 : cluster 0 osdmap e136: 8 total, 7 up, 8 in 2023-12-16T07:51:13.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:13 smithi169 bash[51638]: audit 2023-12-16T07:51:12.563881+0000 mon.a (mon.0) 525 : audit 1 from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:51:13.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:13 smithi169 bash[51638]: audit 2023-12-16T07:51:12.565588+0000 mon.b (mon.2) 8 : audit 1 from='osd.6 [v2:172.21.15.169:6816/2677142792,v1:172.21.15.169:6817/2677142792]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:51:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:13 smithi169 bash[51638]: audit 2023-12-16T07:51:12.604683+0000 mon.a (mon.0) 526 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:13 smithi169 bash[51638]: audit 2023-12-16T07:51:12.615454+0000 mon.a (mon.0) 527 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:13.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:13 smithi169 bash[51638]: audit 2023-12-16T07:51:13.526286+0000 mon.a (mon.0) 528 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:14.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[64479]: cluster 2023-12-16T07:51:13.275050+0000 mgr.y (mgr.25477) 315 : cluster 0 pgmap v196: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2023-12-16T07:51:14.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[64479]: cluster 2023-12-16T07:51:13.601065+0000 mon.a (mon.0) 529 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:51:14.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[64479]: cluster 2023-12-16T07:51:13.610254+0000 mon.a (mon.0) 530 : cluster 1 osd.6 [v2:172.21.15.169:6816/2677142792,v1:172.21.15.169:6817/2677142792] boot 2023-12-16T07:51:14.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[64479]: cluster 2023-12-16T07:51:13.610340+0000 mon.a (mon.0) 531 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2023-12-16T07:51:14.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[64479]: cluster 2023-12-16T07:51:13.613869+0000 osd.7 (osd.7) 55 : cluster 3 failed to encode map e137 with expected crc 2023-12-16T07:51:14.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[64479]: audit 2023-12-16T07:51:13.614916+0000 mon.a (mon.0) 532 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:51:14.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[64479]: cluster 2023-12-16T07:51:13.623711+0000 osd.7 (osd.7) 56 : cluster 3 failed to encode map e137 with expected crc 2023-12-16T07:51:14.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[64479]: cluster 2023-12-16T07:51:13.625141+0000 osd.7 (osd.7) 57 : cluster 3 failed to encode map e137 with expected crc 2023-12-16T07:51:14.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[65864]: cluster 2023-12-16T07:51:13.275050+0000 mgr.y (mgr.25477) 315 : cluster 0 pgmap v196: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2023-12-16T07:51:14.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[65864]: cluster 2023-12-16T07:51:13.601065+0000 mon.a (mon.0) 529 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:51:14.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[65864]: cluster 2023-12-16T07:51:13.610254+0000 mon.a (mon.0) 530 : cluster 1 osd.6 [v2:172.21.15.169:6816/2677142792,v1:172.21.15.169:6817/2677142792] boot 2023-12-16T07:51:14.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[65864]: cluster 2023-12-16T07:51:13.610340+0000 mon.a (mon.0) 531 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2023-12-16T07:51:14.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[65864]: cluster 2023-12-16T07:51:13.613869+0000 osd.7 (osd.7) 55 : cluster 3 failed to encode map e137 with expected crc 2023-12-16T07:51:14.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[65864]: audit 2023-12-16T07:51:13.614916+0000 mon.a (mon.0) 532 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:51:14.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[65864]: cluster 2023-12-16T07:51:13.623711+0000 osd.7 (osd.7) 56 : cluster 3 failed to encode map e137 with expected crc 2023-12-16T07:51:14.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:14 smithi007 bash[65864]: cluster 2023-12-16T07:51:13.625141+0000 osd.7 (osd.7) 57 : cluster 3 failed to encode map e137 with expected crc 2023-12-16T07:51:14.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:14 smithi169 bash[51638]: cluster 2023-12-16T07:51:13.275050+0000 mgr.y (mgr.25477) 315 : cluster 0 pgmap v196: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 76/630 objects degraded (12.063%) 2023-12-16T07:51:14.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:14 smithi169 bash[51638]: cluster 2023-12-16T07:51:13.601065+0000 mon.a (mon.0) 529 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:51:14.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:14 smithi169 bash[51638]: cluster 2023-12-16T07:51:13.610254+0000 mon.a (mon.0) 530 : cluster 1 osd.6 [v2:172.21.15.169:6816/2677142792,v1:172.21.15.169:6817/2677142792] boot 2023-12-16T07:51:14.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:14 smithi169 bash[51638]: cluster 2023-12-16T07:51:13.610340+0000 mon.a (mon.0) 531 : cluster 0 osdmap e137: 8 total, 8 up, 8 in 2023-12-16T07:51:14.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:14 smithi169 bash[51638]: cluster 2023-12-16T07:51:13.613869+0000 osd.7 (osd.7) 55 : cluster 3 failed to encode map e137 with expected crc 2023-12-16T07:51:14.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:14 smithi169 bash[51638]: audit 2023-12-16T07:51:13.614916+0000 mon.a (mon.0) 532 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-12-16T07:51:14.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:14 smithi169 bash[51638]: cluster 2023-12-16T07:51:13.623711+0000 osd.7 (osd.7) 56 : cluster 3 failed to encode map e137 with expected crc 2023-12-16T07:51:14.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:14 smithi169 bash[51638]: cluster 2023-12-16T07:51:13.625141+0000 osd.7 (osd.7) 57 : cluster 3 failed to encode map e137 with expected crc 2023-12-16T07:51:15.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:15 smithi169 bash[51638]: audit 2023-12-16T07:51:14.363431+0000 mgr.y (mgr.25477) 316 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:15.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:15 smithi169 bash[51638]: cluster 2023-12-16T07:51:14.609812+0000 osd.7 (osd.7) 58 : cluster 3 failed to encode map e138 with expected crc 2023-12-16T07:51:15.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:15 smithi169 bash[51638]: cluster 2023-12-16T07:51:14.612706+0000 mon.a (mon.0) 533 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2023-12-16T07:51:15.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:15 smithi169 bash[51638]: cluster 2023-12-16T07:51:14.613751+0000 osd.7 (osd.7) 59 : cluster 3 failed to encode map e138 with expected crc 2023-12-16T07:51:16.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:15 smithi007 bash[64479]: audit 2023-12-16T07:51:14.363431+0000 mgr.y (mgr.25477) 316 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:16.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:15 smithi007 bash[64479]: cluster 2023-12-16T07:51:14.609812+0000 osd.7 (osd.7) 58 : cluster 3 failed to encode map e138 with expected crc 2023-12-16T07:51:16.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:15 smithi007 bash[64479]: cluster 2023-12-16T07:51:14.612706+0000 mon.a (mon.0) 533 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2023-12-16T07:51:16.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:15 smithi007 bash[64479]: cluster 2023-12-16T07:51:14.613751+0000 osd.7 (osd.7) 59 : cluster 3 failed to encode map e138 with expected crc 2023-12-16T07:51:16.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:15 smithi007 bash[65864]: audit 2023-12-16T07:51:14.363431+0000 mgr.y (mgr.25477) 316 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:16.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:15 smithi007 bash[65864]: cluster 2023-12-16T07:51:14.609812+0000 osd.7 (osd.7) 58 : cluster 3 failed to encode map e138 with expected crc 2023-12-16T07:51:16.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:15 smithi007 bash[65864]: cluster 2023-12-16T07:51:14.612706+0000 mon.a (mon.0) 533 : cluster 0 osdmap e138: 8 total, 8 up, 8 in 2023-12-16T07:51:16.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:15 smithi007 bash[65864]: cluster 2023-12-16T07:51:14.613751+0000 osd.7 (osd.7) 59 : cluster 3 failed to encode map e138 with expected crc 2023-12-16T07:51:16.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:16 smithi169 bash[51638]: cluster 2023-12-16T07:51:15.276013+0000 mgr.y (mgr.25477) 317 : cluster 0 pgmap v199: 161 pgs: 20 peering, 21 active+undersized, 12 active+undersized+degraded, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 48/630 objects degraded (7.619%) 2023-12-16T07:51:16.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:16 smithi169 bash[51638]: cluster 2023-12-16T07:51:15.605611+0000 mon.a (mon.0) 534 : cluster 3 Health check update: Degraded data redundancy: 48/630 objects degraded (7.619%), 12 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:17.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:16 smithi007 bash[65864]: cluster 2023-12-16T07:51:15.276013+0000 mgr.y (mgr.25477) 317 : cluster 0 pgmap v199: 161 pgs: 20 peering, 21 active+undersized, 12 active+undersized+degraded, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 48/630 objects degraded (7.619%) 2023-12-16T07:51:17.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:16 smithi007 bash[65864]: cluster 2023-12-16T07:51:15.605611+0000 mon.a (mon.0) 534 : cluster 3 Health check update: Degraded data redundancy: 48/630 objects degraded (7.619%), 12 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:17.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:16 smithi007 bash[64479]: cluster 2023-12-16T07:51:15.276013+0000 mgr.y (mgr.25477) 317 : cluster 0 pgmap v199: 161 pgs: 20 peering, 21 active+undersized, 12 active+undersized+degraded, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 48/630 objects degraded (7.619%) 2023-12-16T07:51:17.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:16 smithi007 bash[64479]: cluster 2023-12-16T07:51:15.605611+0000 mon.a (mon.0) 534 : cluster 3 Health check update: Degraded data redundancy: 48/630 objects degraded (7.619%), 12 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:17.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:17 smithi169 bash[51638]: cluster 2023-12-16T07:51:17.277115+0000 mgr.y (mgr.25477) 318 : cluster 0 pgmap v200: 161 pgs: 20 peering, 19 active+undersized, 10 active+undersized+degraded, 112 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 38/630 objects degraded (6.032%) 2023-12-16T07:51:18.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:17 smithi007 bash[65864]: cluster 2023-12-16T07:51:17.277115+0000 mgr.y (mgr.25477) 318 : cluster 0 pgmap v200: 161 pgs: 20 peering, 19 active+undersized, 10 active+undersized+degraded, 112 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 38/630 objects degraded (6.032%) 2023-12-16T07:51:18.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:17 smithi007 bash[64479]: cluster 2023-12-16T07:51:17.277115+0000 mgr.y (mgr.25477) 318 : cluster 0 pgmap v200: 161 pgs: 20 peering, 19 active+undersized, 10 active+undersized+degraded, 112 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 38/630 objects degraded (6.032%) 2023-12-16T07:51:20.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:19 smithi169 bash[51638]: cluster 2023-12-16T07:51:19.278489+0000 mgr.y (mgr.25477) 319 : cluster 0 pgmap v201: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 609 B/s rd, 0 op/s 2023-12-16T07:51:20.250 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:19 smithi007 bash[65864]: cluster 2023-12-16T07:51:19.278489+0000 mgr.y (mgr.25477) 319 : cluster 0 pgmap v201: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 609 B/s rd, 0 op/s 2023-12-16T07:51:20.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:19 smithi007 bash[64479]: cluster 2023-12-16T07:51:19.278489+0000 mgr.y (mgr.25477) 319 : cluster 0 pgmap v201: 161 pgs: 11 peering, 150 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 609 B/s rd, 0 op/s 2023-12-16T07:51:20.567 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:51:20 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:51:20] "GET /metrics HTTP/1.1" 200 38427 "" "Prometheus/2.33.4" 2023-12-16T07:51:20.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:20 smithi169 bash[51638]: cluster 2023-12-16T07:51:19.880745+0000 mon.a (mon.0) 535 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 10 pgs degraded) 2023-12-16T07:51:20.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:20 smithi169 bash[51638]: cluster 2023-12-16T07:51:19.880819+0000 mon.a (mon.0) 536 : cluster 1 Cluster is now healthy 2023-12-16T07:51:21.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:20 smithi007 bash[64479]: cluster 2023-12-16T07:51:19.880745+0000 mon.a (mon.0) 535 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 10 pgs degraded) 2023-12-16T07:51:21.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:20 smithi007 bash[64479]: cluster 2023-12-16T07:51:19.880819+0000 mon.a (mon.0) 536 : cluster 1 Cluster is now healthy 2023-12-16T07:51:21.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:20 smithi007 bash[65864]: cluster 2023-12-16T07:51:19.880745+0000 mon.a (mon.0) 535 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 10 pgs degraded) 2023-12-16T07:51:21.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:20 smithi007 bash[65864]: cluster 2023-12-16T07:51:19.880819+0000 mon.a (mon.0) 536 : cluster 1 Cluster is now healthy 2023-12-16T07:51:22.184 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:21 smithi169 bash[51638]: cluster 2023-12-16T07:51:21.279665+0000 mgr.y (mgr.25477) 320 : cluster 0 pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:51:22.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:21 smithi007 bash[64479]: cluster 2023-12-16T07:51:21.279665+0000 mgr.y (mgr.25477) 320 : cluster 0 pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:51:22.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:21 smithi007 bash[65864]: cluster 2023-12-16T07:51:21.279665+0000 mgr.y (mgr.25477) 320 : cluster 0 pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:51:23.571 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:51:23.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: audit 2023-12-16T07:51:22.621310+0000 mon.a (mon.0) 537 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:23.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: audit 2023-12-16T07:51:22.635126+0000 mon.a (mon.0) 538 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:23.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: audit 2023-12-16T07:51:22.637182+0000 mon.a (mon.0) 539 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:51:23.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: audit 2023-12-16T07:51:22.638899+0000 mon.a (mon.0) 540 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:51:23.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: audit 2023-12-16T07:51:22.647759+0000 mon.a (mon.0) 541 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:23.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: audit 2023-12-16T07:51:22.732964+0000 mon.a (mon.0) 542 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:51:23.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: audit 2023-12-16T07:51:22.736441+0000 mon.a (mon.0) 543 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:23.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: audit 2023-12-16T07:51:22.739162+0000 mon.a (mon.0) 544 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:23.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: audit 2023-12-16T07:51:22.741295+0000 mon.a (mon.0) 545 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:23.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: audit 2023-12-16T07:51:22.744054+0000 mon.a (mon.0) 546 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-12-16T07:51:23.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: audit 2023-12-16T07:51:22.744421+0000 mgr.y (mgr.25477) 321 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-12-16T07:51:23.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: cephadm 2023-12-16T07:51:22.746533+0000 mgr.y (mgr.25477) 322 : cephadm 1 Upgrade: osd.7 is safe to restart 2023-12-16T07:51:23.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: audit 2023-12-16T07:51:23.213131+0000 mon.a (mon.0) 547 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:23.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: audit 2023-12-16T07:51:23.214659+0000 mon.a (mon.0) 548 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:51:23.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: cluster 2023-12-16T07:51:23.280884+0000 mgr.y (mgr.25477) 323 : cluster 0 pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2023-12-16T07:51:23.935 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:23 smithi169 bash[51638]: audit 2023-12-16T07:51:23.553528+0000 mgr.y (mgr.25477) 324 : audit 0 from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:24.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: audit 2023-12-16T07:51:22.621310+0000 mon.a (mon.0) 537 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:24.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: audit 2023-12-16T07:51:22.635126+0000 mon.a (mon.0) 538 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:24.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: audit 2023-12-16T07:51:22.637182+0000 mon.a (mon.0) 539 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:51:24.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: audit 2023-12-16T07:51:22.638899+0000 mon.a (mon.0) 540 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:51:24.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: audit 2023-12-16T07:51:22.647759+0000 mon.a (mon.0) 541 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:24.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: audit 2023-12-16T07:51:22.732964+0000 mon.a (mon.0) 542 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:51:24.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: audit 2023-12-16T07:51:22.736441+0000 mon.a (mon.0) 543 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:24.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: audit 2023-12-16T07:51:22.739162+0000 mon.a (mon.0) 544 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:24.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: audit 2023-12-16T07:51:22.741295+0000 mon.a (mon.0) 545 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:24.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: audit 2023-12-16T07:51:22.744054+0000 mon.a (mon.0) 546 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-12-16T07:51:24.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: audit 2023-12-16T07:51:22.744421+0000 mgr.y (mgr.25477) 321 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-12-16T07:51:24.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: cephadm 2023-12-16T07:51:22.746533+0000 mgr.y (mgr.25477) 322 : cephadm 1 Upgrade: osd.7 is safe to restart 2023-12-16T07:51:24.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: audit 2023-12-16T07:51:23.213131+0000 mon.a (mon.0) 547 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:24.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: audit 2023-12-16T07:51:23.214659+0000 mon.a (mon.0) 548 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:51:24.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: cluster 2023-12-16T07:51:23.280884+0000 mgr.y (mgr.25477) 323 : cluster 0 pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2023-12-16T07:51:24.069 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[65864]: audit 2023-12-16T07:51:23.553528+0000 mgr.y (mgr.25477) 324 : audit 0 from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:24.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: audit 2023-12-16T07:51:22.621310+0000 mon.a (mon.0) 537 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:24.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: audit 2023-12-16T07:51:22.635126+0000 mon.a (mon.0) 538 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:24.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: audit 2023-12-16T07:51:22.637182+0000 mon.a (mon.0) 539 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:51:24.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: audit 2023-12-16T07:51:22.638899+0000 mon.a (mon.0) 540 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:51:24.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: audit 2023-12-16T07:51:22.647759+0000 mon.a (mon.0) 541 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:24.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: audit 2023-12-16T07:51:22.732964+0000 mon.a (mon.0) 542 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:51:24.069 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: audit 2023-12-16T07:51:22.736441+0000 mon.a (mon.0) 543 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:24.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: audit 2023-12-16T07:51:22.739162+0000 mon.a (mon.0) 544 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:24.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: audit 2023-12-16T07:51:22.741295+0000 mon.a (mon.0) 545 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:24.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: audit 2023-12-16T07:51:22.744054+0000 mon.a (mon.0) 546 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-12-16T07:51:24.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: audit 2023-12-16T07:51:22.744421+0000 mgr.y (mgr.25477) 321 : audit 0 from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2023-12-16T07:51:24.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: cephadm 2023-12-16T07:51:22.746533+0000 mgr.y (mgr.25477) 322 : cephadm 1 Upgrade: osd.7 is safe to restart 2023-12-16T07:51:24.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: audit 2023-12-16T07:51:23.213131+0000 mon.a (mon.0) 547 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:24.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: audit 2023-12-16T07:51:23.214659+0000 mon.a (mon.0) 548 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:51:24.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: cluster 2023-12-16T07:51:23.280884+0000 mgr.y (mgr.25477) 323 : cluster 0 pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2023-12-16T07:51:24.070 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:23 smithi007 bash[64479]: audit 2023-12-16T07:51:23.553528+0000 mgr.y (mgr.25477) 324 : audit 0 from='client.34530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:24.342 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:51:24.342 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (7m) 100s ago 14m 16.8M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:51:24.342 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (7m) 12s ago 14m 44.3M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:51:24.342 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (6m) 100s ago 12m 47.7M - 3.5 e1d6a67b021e 8d8009d0c83f 2023-12-16T07:51:24.342 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283,8765 running (6m) 12s ago 20m 436M - 19.0.0-191-g6d933094 a29c258b5d6f 5295ba35c2cd 2023-12-16T07:51:24.342 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (7m) 100s ago 23m 493M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:51:24.342 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (5m) 100s ago 23m 52.7M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 5a266b45f3ce 2023-12-16T07:51:24.342 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (4m) 12s ago 21m 45.2M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1fc9bf864dda 2023-12-16T07:51:24.342 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (5m) 100s ago 21m 42.0M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1dc91c31fc27 2023-12-16T07:51:24.343 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (8m) 100s ago 15m 13.0M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:51:24.343 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (7m) 12s ago 15m 12.5M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:51:24.343 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (3m) 100s ago 20m 52.0M 2782M 19.0.0-191-g6d933094 a29c258b5d6f a00724c6d077 2023-12-16T07:51:24.343 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (2m) 100s ago 19m 51.3M 2782M 19.0.0-191-g6d933094 a29c258b5d6f 5e90fce88079 2023-12-16T07:51:24.343 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (2m) 100s ago 19m 46.9M 2782M 19.0.0-191-g6d933094 a29c258b5d6f 906a30291e88 2023-12-16T07:51:24.343 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (105s) 100s ago 18m - 2782M 19.0.0-191-g6d933094 a29c258b5d6f 8e5e7d6ede0f 2023-12-16T07:51:24.343 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (76s) 12s ago 18m 50.8M 3037M 19.0.0-191-g6d933094 a29c258b5d6f b612e0176579 2023-12-16T07:51:24.343 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (46s) 12s ago 17m 48.6M 3037M 19.0.0-191-g6d933094 a29c258b5d6f c1ca50cfe550 2023-12-16T07:51:24.343 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (16s) 12s ago 16m - 3037M 19.0.0-191-g6d933094 a29c258b5d6f b335dc393bbe 2023-12-16T07:51:24.343 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (16m) 12s ago 16m 68.1M 3037M 17.2.0 e1d6a67b021e cb5540434ba4 2023-12-16T07:51:24.343 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (6m) 12s ago 15m 64.2M - 2.33.4 514e6a882f6e 2652ce1b8540 2023-12-16T07:51:24.343 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (13m) 100s ago 13m 88.1M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:51:24.343 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (13m) 12s ago 13m 88.8M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:51:24.343 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (12m) 100s ago 12m 88.0M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:51:24.344 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (12m) 12s ago 12m 88.6M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:51:24.798 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:51:24.798 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:51:24.798 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 3 2023-12-16T07:51:24.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:51:24.798 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:51:24.798 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:51:24.798 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:51:24.798 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:51:24.798 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2023-12-16T07:51:24.799 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 7 2023-12-16T07:51:24.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:51:24.799 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:51:24.799 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:51:24.799 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:51:24.799 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:51:24.799 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2023-12-16T07:51:24.799 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 12 2023-12-16T07:51:24.799 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:51:24.799 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:51:25.208 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:51:25.208 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:51:25.208 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:51:25.208 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:51:25.208 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2023-12-16T07:51:25.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr", 2023-12-16T07:51:25.208 INFO:teuthology.orchestra.run.smithi007.stdout: "mon" 2023-12-16T07:51:25.209 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:51:25.209 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "12/23 daemons upgraded", 2023-12-16T07:51:25.209 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading osd daemons", 2023-12-16T07:51:25.209 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2023-12-16T07:51:25.209 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:51:25.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:25 smithi007 bash[65864]: audit 2023-12-16T07:51:23.937769+0000 mgr.y (mgr.25477) 325 : audit 0 from='client.44464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:25.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:25 smithi007 bash[65864]: audit 2023-12-16T07:51:24.315103+0000 mgr.y (mgr.25477) 326 : audit 0 from='client.44470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:25.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:25 smithi007 bash[65864]: audit 2023-12-16T07:51:24.371362+0000 mgr.y (mgr.25477) 327 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:25.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:25 smithi007 bash[65864]: audit 2023-12-16T07:51:24.794968+0000 mon.a (mon.0) 549 : audit 0 from='client.? 172.21.15.7:0/146203515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:25.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:25 smithi007 bash[65864]: audit 2023-12-16T07:51:25.023292+0000 mon.a (mon.0) 550 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:25.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:25 smithi007 bash[65864]: audit 2023-12-16T07:51:25.027704+0000 mon.a (mon.0) 551 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-12-16T07:51:25.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:25 smithi007 bash[65864]: audit 2023-12-16T07:51:25.028624+0000 mon.a (mon.0) 552 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:51:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:25 smithi007 bash[64479]: audit 2023-12-16T07:51:23.937769+0000 mgr.y (mgr.25477) 325 : audit 0 from='client.44464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:25 smithi007 bash[64479]: audit 2023-12-16T07:51:24.315103+0000 mgr.y (mgr.25477) 326 : audit 0 from='client.44470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:25 smithi007 bash[64479]: audit 2023-12-16T07:51:24.371362+0000 mgr.y (mgr.25477) 327 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:25 smithi007 bash[64479]: audit 2023-12-16T07:51:24.794968+0000 mon.a (mon.0) 549 : audit 0 from='client.? 172.21.15.7:0/146203515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:25.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:25 smithi007 bash[64479]: audit 2023-12-16T07:51:25.023292+0000 mon.a (mon.0) 550 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:25.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:25 smithi007 bash[64479]: audit 2023-12-16T07:51:25.027704+0000 mon.a (mon.0) 551 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-12-16T07:51:25.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:25 smithi007 bash[64479]: audit 2023-12-16T07:51:25.028624+0000 mon.a (mon.0) 552 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:51:25.671 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2023-12-16T07:51:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:25 smithi169 bash[51638]: audit 2023-12-16T07:51:23.937769+0000 mgr.y (mgr.25477) 325 : audit 0 from='client.44464 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:25 smithi169 bash[51638]: audit 2023-12-16T07:51:24.315103+0000 mgr.y (mgr.25477) 326 : audit 0 from='client.44470 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:25 smithi169 bash[51638]: audit 2023-12-16T07:51:24.371362+0000 mgr.y (mgr.25477) 327 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:25 smithi169 bash[51638]: audit 2023-12-16T07:51:24.794968+0000 mon.a (mon.0) 549 : audit 0 from='client.? 172.21.15.7:0/146203515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:25 smithi169 bash[51638]: audit 2023-12-16T07:51:25.023292+0000 mon.a (mon.0) 550 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:25.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:25 smithi169 bash[51638]: audit 2023-12-16T07:51:25.027704+0000 mon.a (mon.0) 551 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-12-16T07:51:25.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:25 smithi169 bash[51638]: audit 2023-12-16T07:51:25.028624+0000 mon.a (mon.0) 552 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:51:26.495 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:26 smithi169 bash[51638]: cephadm 2023-12-16T07:51:25.015853+0000 mgr.y (mgr.25477) 328 : cephadm 1 Upgrade: Updating osd.7 2023-12-16T07:51:26.495 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:26 smithi169 bash[51638]: cephadm 2023-12-16T07:51:25.032109+0000 mgr.y (mgr.25477) 329 : cephadm 1 Deploying daemon osd.7 on smithi169 2023-12-16T07:51:26.495 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:26 smithi169 bash[51638]: audit 2023-12-16T07:51:25.204305+0000 mgr.y (mgr.25477) 330 : audit 0 from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:26.495 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:26 smithi169 bash[51638]: cluster 2023-12-16T07:51:25.282310+0000 mgr.y (mgr.25477) 331 : cluster 0 pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-12-16T07:51:26.495 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:26 smithi169 bash[51638]: audit 2023-12-16T07:51:25.668646+0000 mon.a (mon.0) 553 : audit 0 from='client.? 172.21.15.7:0/4188621445' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:51:26.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:26 smithi007 bash[64479]: cephadm 2023-12-16T07:51:25.015853+0000 mgr.y (mgr.25477) 328 : cephadm 1 Upgrade: Updating osd.7 2023-12-16T07:51:26.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:26 smithi007 bash[64479]: cephadm 2023-12-16T07:51:25.032109+0000 mgr.y (mgr.25477) 329 : cephadm 1 Deploying daemon osd.7 on smithi169 2023-12-16T07:51:26.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:26 smithi007 bash[64479]: audit 2023-12-16T07:51:25.204305+0000 mgr.y (mgr.25477) 330 : audit 0 from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:26.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:26 smithi007 bash[64479]: cluster 2023-12-16T07:51:25.282310+0000 mgr.y (mgr.25477) 331 : cluster 0 pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-12-16T07:51:26.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:26 smithi007 bash[64479]: audit 2023-12-16T07:51:25.668646+0000 mon.a (mon.0) 553 : audit 0 from='client.? 172.21.15.7:0/4188621445' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:51:26.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:26 smithi007 bash[65864]: cephadm 2023-12-16T07:51:25.015853+0000 mgr.y (mgr.25477) 328 : cephadm 1 Upgrade: Updating osd.7 2023-12-16T07:51:26.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:26 smithi007 bash[65864]: cephadm 2023-12-16T07:51:25.032109+0000 mgr.y (mgr.25477) 329 : cephadm 1 Deploying daemon osd.7 on smithi169 2023-12-16T07:51:26.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:26 smithi007 bash[65864]: audit 2023-12-16T07:51:25.204305+0000 mgr.y (mgr.25477) 330 : audit 0 from='client.34554 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:26.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:26 smithi007 bash[65864]: cluster 2023-12-16T07:51:25.282310+0000 mgr.y (mgr.25477) 331 : cluster 0 pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-12-16T07:51:26.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:26 smithi007 bash[65864]: audit 2023-12-16T07:51:25.668646+0000 mon.a (mon.0) 553 : audit 0 from='client.? 172.21.15.7:0/4188621445' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:51:28.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:28 smithi169 bash[51638]: cluster 2023-12-16T07:51:27.283238+0000 mgr.y (mgr.25477) 332 : cluster 0 pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-12-16T07:51:28.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:28 smithi007 bash[64479]: cluster 2023-12-16T07:51:27.283238+0000 mgr.y (mgr.25477) 332 : cluster 0 pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-12-16T07:51:28.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:28 smithi007 bash[65864]: cluster 2023-12-16T07:51:27.283238+0000 mgr.y (mgr.25477) 332 : cluster 0 pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-12-16T07:51:29.551 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:29 smithi169 systemd[1]: Stopping Ceph osd.7 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:51:29.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:29 smithi007 bash[65864]: audit 2023-12-16T07:51:28.551004+0000 mon.a (mon.0) 554 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:29.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:29 smithi007 bash[64479]: audit 2023-12-16T07:51:28.551004+0000 mon.a (mon.0) 554 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:29.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:29 smithi169 bash[51638]: audit 2023-12-16T07:51:28.551004+0000 mon.a (mon.0) 554 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:29.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:29 smithi169 bash[37512]: debug 2023-12-16T07:51:29.638+0000 7f74a48fb700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:51:29.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:29 smithi169 bash[37512]: debug 2023-12-16T07:51:29.638+0000 7f74a48fb700 -1 osd.7 138 *** Got signal Terminated *** 2023-12-16T07:51:29.934 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:29 smithi169 bash[37512]: debug 2023-12-16T07:51:29.638+0000 7f74a48fb700 -1 osd.7 138 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:51:30.554 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:51:30 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:51:30] "GET /metrics HTTP/1.1" 200 38431 "" "Prometheus/2.33.4" 2023-12-16T07:51:30.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:30 smithi007 bash[65864]: cluster 2023-12-16T07:51:29.284567+0000 mgr.y (mgr.25477) 333 : cluster 0 pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-12-16T07:51:30.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:30 smithi007 bash[65864]: cluster 2023-12-16T07:51:29.644730+0000 mon.a (mon.0) 555 : cluster 1 osd.7 marked itself down and dead 2023-12-16T07:51:30.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:30 smithi007 bash[64479]: cluster 2023-12-16T07:51:29.284567+0000 mgr.y (mgr.25477) 333 : cluster 0 pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-12-16T07:51:30.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:30 smithi007 bash[64479]: cluster 2023-12-16T07:51:29.644730+0000 mon.a (mon.0) 555 : cluster 1 osd.7 marked itself down and dead 2023-12-16T07:51:30.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:30 smithi169 bash[51638]: cluster 2023-12-16T07:51:29.284567+0000 mgr.y (mgr.25477) 333 : cluster 0 pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2023-12-16T07:51:30.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:30 smithi169 bash[51638]: cluster 2023-12-16T07:51:29.644730+0000 mon.a (mon.0) 555 : cluster 1 osd.7 marked itself down and dead 2023-12-16T07:51:31.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:31 smithi007 bash[65864]: cluster 2023-12-16T07:51:30.550464+0000 mon.a (mon.0) 556 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:51:31.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:31 smithi007 bash[65864]: cluster 2023-12-16T07:51:30.565483+0000 mon.a (mon.0) 557 : cluster 0 osdmap e139: 8 total, 7 up, 8 in 2023-12-16T07:51:31.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:31 smithi007 bash[64479]: cluster 2023-12-16T07:51:30.550464+0000 mon.a (mon.0) 556 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:51:31.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:31 smithi007 bash[64479]: cluster 2023-12-16T07:51:30.565483+0000 mon.a (mon.0) 557 : cluster 0 osdmap e139: 8 total, 7 up, 8 in 2023-12-16T07:51:31.823 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:31 smithi169 bash[51638]: cluster 2023-12-16T07:51:30.550464+0000 mon.a (mon.0) 556 : cluster 3 Health check failed: 1 osds down (OSD_DOWN) 2023-12-16T07:51:31.823 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:31 smithi169 bash[51638]: cluster 2023-12-16T07:51:30.565483+0000 mon.a (mon.0) 557 : cluster 0 osdmap e139: 8 total, 7 up, 8 in 2023-12-16T07:51:31.823 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:31 smithi169 bash[59600]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-7 2023-12-16T07:51:31.823 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:31 smithi169 bash[59674]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-7 2023-12-16T07:51:32.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:32 smithi007 bash[64479]: cluster 2023-12-16T07:51:31.285569+0000 mgr.y (mgr.25477) 334 : cluster 0 pgmap v208: 161 pgs: 4 peering, 22 stale+active+clean, 135 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:51:32.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:32 smithi007 bash[64479]: cluster 2023-12-16T07:51:31.562116+0000 mon.a (mon.0) 558 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2023-12-16T07:51:32.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:32 smithi007 bash[64479]: cluster 2023-12-16T07:51:31.571109+0000 mon.a (mon.0) 559 : cluster 0 osdmap e140: 8 total, 7 up, 8 in 2023-12-16T07:51:32.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:32 smithi007 bash[65864]: cluster 2023-12-16T07:51:31.285569+0000 mgr.y (mgr.25477) 334 : cluster 0 pgmap v208: 161 pgs: 4 peering, 22 stale+active+clean, 135 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:51:32.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:32 smithi007 bash[65864]: cluster 2023-12-16T07:51:31.562116+0000 mon.a (mon.0) 558 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2023-12-16T07:51:32.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:32 smithi007 bash[65864]: cluster 2023-12-16T07:51:31.571109+0000 mon.a (mon.0) 559 : cluster 0 osdmap e140: 8 total, 7 up, 8 in 2023-12-16T07:51:32.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:32 smithi169 bash[51638]: cluster 2023-12-16T07:51:31.285569+0000 mgr.y (mgr.25477) 334 : cluster 0 pgmap v208: 161 pgs: 4 peering, 22 stale+active+clean, 135 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:51:32.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:32 smithi169 bash[51638]: cluster 2023-12-16T07:51:31.562116+0000 mon.a (mon.0) 558 : cluster 3 Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2023-12-16T07:51:32.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:32 smithi169 bash[51638]: cluster 2023-12-16T07:51:31.571109+0000 mon.a (mon.0) 559 : cluster 0 osdmap e140: 8 total, 7 up, 8 in 2023-12-16T07:51:33.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:33 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.7.service: Succeeded. 2023-12-16T07:51:33.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:33 smithi169 systemd[1]: Stopped Ceph osd.7 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:51:33.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:33 smithi169 bash[51638]: cluster 2023-12-16T07:51:33.286925+0000 mgr.y (mgr.25477) 335 : cluster 0 pgmap v210: 161 pgs: 13 active+undersized, 9 active+undersized+degraded, 4 peering, 10 stale+active+clean, 125 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 36/630 objects degraded (5.714%) 2023-12-16T07:51:33.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:33 smithi169 bash[51638]: cluster 2023-12-16T07:51:33.555610+0000 mon.a (mon.0) 560 : cluster 3 Health check failed: Degraded data redundancy: 36/630 objects degraded (5.714%), 9 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:34.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:33 smithi007 bash[65864]: cluster 2023-12-16T07:51:33.286925+0000 mgr.y (mgr.25477) 335 : cluster 0 pgmap v210: 161 pgs: 13 active+undersized, 9 active+undersized+degraded, 4 peering, 10 stale+active+clean, 125 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 36/630 objects degraded (5.714%) 2023-12-16T07:51:34.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:33 smithi007 bash[65864]: cluster 2023-12-16T07:51:33.555610+0000 mon.a (mon.0) 560 : cluster 3 Health check failed: Degraded data redundancy: 36/630 objects degraded (5.714%), 9 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:34.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:33 smithi007 bash[64479]: cluster 2023-12-16T07:51:33.286925+0000 mgr.y (mgr.25477) 335 : cluster 0 pgmap v210: 161 pgs: 13 active+undersized, 9 active+undersized+degraded, 4 peering, 10 stale+active+clean, 125 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 36/630 objects degraded (5.714%) 2023-12-16T07:51:34.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:33 smithi007 bash[64479]: cluster 2023-12-16T07:51:33.555610+0000 mon.a (mon.0) 560 : cluster 3 Health check failed: Degraded data redundancy: 36/630 objects degraded (5.714%), 9 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:34.433 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:34 smithi169 systemd[1]: Started Ceph osd.7 for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:51:35.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:35 smithi169 bash[51638]: audit 2023-12-16T07:51:34.082742+0000 mon.a (mon.0) 561 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:35.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:35 smithi169 bash[51638]: audit 2023-12-16T07:51:34.091826+0000 mon.a (mon.0) 562 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:35.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:35 smithi169 bash[51638]: audit 2023-12-16T07:51:34.382455+0000 mgr.y (mgr.25477) 336 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:35.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:35 smithi007 bash[65864]: audit 2023-12-16T07:51:34.082742+0000 mon.a (mon.0) 561 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:35.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:35 smithi007 bash[65864]: audit 2023-12-16T07:51:34.091826+0000 mon.a (mon.0) 562 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:35.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:35 smithi007 bash[65864]: audit 2023-12-16T07:51:34.382455+0000 mgr.y (mgr.25477) 336 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:35.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:35 smithi007 bash[64479]: audit 2023-12-16T07:51:34.082742+0000 mon.a (mon.0) 561 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:35.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:35 smithi007 bash[64479]: audit 2023-12-16T07:51:34.091826+0000 mon.a (mon.0) 562 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:35.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:35 smithi007 bash[64479]: audit 2023-12-16T07:51:34.382455+0000 mgr.y (mgr.25477) 336 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:35.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:35 smithi169 bash[59841]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-16T07:51:35.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:35 smithi169 bash[59841]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2023-12-16T07:51:35.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:35 smithi169 bash[59841]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2023-12-16T07:51:35.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:35 smithi169 bash[59841]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2023-12-16T07:51:35.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:35 smithi169 bash[59841]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2023-12-16T07:51:35.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:35 smithi169 bash[59841]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-16T07:51:35.934 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:35 smithi169 bash[59841]: --> ceph-volume raw activate successful for osd ID: 7 2023-12-16T07:51:36.361 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:36 smithi169 bash[51638]: cluster 2023-12-16T07:51:35.288631+0000 mgr.y (mgr.25477) 337 : cluster 0 pgmap v211: 161 pgs: 40 active+undersized, 25 active+undersized+degraded, 4 peering, 92 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2023-12-16T07:51:36.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:36 smithi007 bash[64479]: cluster 2023-12-16T07:51:35.288631+0000 mgr.y (mgr.25477) 337 : cluster 0 pgmap v211: 161 pgs: 40 active+undersized, 25 active+undersized+degraded, 4 peering, 92 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2023-12-16T07:51:36.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:36 smithi007 bash[65864]: cluster 2023-12-16T07:51:35.288631+0000 mgr.y (mgr.25477) 337 : cluster 0 pgmap v211: 161 pgs: 40 active+undersized, 25 active+undersized+degraded, 4 peering, 92 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 81/630 objects degraded (12.857%) 2023-12-16T07:51:37.398 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:37 smithi169 bash[60136]: debug 2023-12-16T07:51:37.234+0000 7f50680d2700 1 -- 172.21.15.169:0/4267299200 <== mon.2 v2:172.21.15.169:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561c093e16c0 con 0x561c09466800 2023-12-16T07:51:37.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:37 smithi169 bash[51638]: cluster 2023-12-16T07:51:37.342001+0000 mon.a (mon.0) 563 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2023-12-16T07:51:37.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:37 smithi007 bash[64479]: cluster 2023-12-16T07:51:37.342001+0000 mon.a (mon.0) 563 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2023-12-16T07:51:37.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:37 smithi007 bash[65864]: cluster 2023-12-16T07:51:37.342001+0000 mon.a (mon.0) 563 : cluster 1 Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2023-12-16T07:51:38.183 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:37 smithi169 bash[60136]: debug 2023-12-16T07:51:37.794+0000 7f5071258640 -1 Falling back to public interface 2023-12-16T07:51:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:38 smithi169 bash[51638]: cluster 2023-12-16T07:51:37.289891+0000 mgr.y (mgr.25477) 338 : cluster 0 pgmap v212: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:51:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:38 smithi169 bash[51638]: audit 2023-12-16T07:51:38.213721+0000 mon.a (mon.0) 564 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:38.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:38 smithi169 bash[51638]: audit 2023-12-16T07:51:38.216099+0000 mon.a (mon.0) 565 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:51:38.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:38 smithi007 bash[64479]: cluster 2023-12-16T07:51:37.289891+0000 mgr.y (mgr.25477) 338 : cluster 0 pgmap v212: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:51:38.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:38 smithi007 bash[64479]: audit 2023-12-16T07:51:38.213721+0000 mon.a (mon.0) 564 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:38.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:38 smithi007 bash[64479]: audit 2023-12-16T07:51:38.216099+0000 mon.a (mon.0) 565 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:51:38.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:38 smithi007 bash[65864]: cluster 2023-12-16T07:51:37.289891+0000 mgr.y (mgr.25477) 338 : cluster 0 pgmap v212: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:51:38.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:38 smithi007 bash[65864]: audit 2023-12-16T07:51:38.213721+0000 mon.a (mon.0) 564 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:38.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:38 smithi007 bash[65864]: audit 2023-12-16T07:51:38.216099+0000 mon.a (mon.0) 565 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:51:39.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:39 smithi169 bash[51638]: cluster 2023-12-16T07:51:38.884900+0000 mon.a (mon.0) 566 : cluster 3 Health check update: Degraded data redundancy: 82/630 objects degraded (13.016%), 26 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:39.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:39 smithi007 bash[64479]: cluster 2023-12-16T07:51:38.884900+0000 mon.a (mon.0) 566 : cluster 3 Health check update: Degraded data redundancy: 82/630 objects degraded (13.016%), 26 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:39.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:39 smithi007 bash[65864]: cluster 2023-12-16T07:51:38.884900+0000 mon.a (mon.0) 566 : cluster 3 Health check update: Degraded data redundancy: 82/630 objects degraded (13.016%), 26 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:40.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:40 smithi007 bash[64479]: cluster 2023-12-16T07:51:39.291538+0000 mgr.y (mgr.25477) 339 : cluster 0 pgmap v213: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:51:40.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:51:40 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:51:40] "GET /metrics HTTP/1.1" 200 38438 "" "Prometheus/2.33.4" 2023-12-16T07:51:40.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:40 smithi007 bash[65864]: cluster 2023-12-16T07:51:39.291538+0000 mgr.y (mgr.25477) 339 : cluster 0 pgmap v213: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:51:40.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:40 smithi169 bash[51638]: cluster 2023-12-16T07:51:39.291538+0000 mgr.y (mgr.25477) 339 : cluster 0 pgmap v213: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:51:41.183 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:40 smithi169 bash[60136]: debug 2023-12-16T07:51:40.822+0000 7f5071258640 -1 osd.7 0 read_superblock omap replica is missing. 2023-12-16T07:51:41.183 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:40 smithi169 bash[60136]: debug 2023-12-16T07:51:40.862+0000 7f5071258640 -1 osd.7 138 log_to_monitors true 2023-12-16T07:51:41.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:41 smithi007 bash[64479]: audit 2023-12-16T07:51:40.516994+0000 mon.a (mon.0) 567 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:41.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:41 smithi007 bash[64479]: audit 2023-12-16T07:51:40.527587+0000 mon.a (mon.0) 568 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:41.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:41 smithi007 bash[64479]: audit 2023-12-16T07:51:40.866570+0000 mon.a (mon.0) 569 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-12-16T07:51:41.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:41 smithi007 bash[64479]: audit 2023-12-16T07:51:40.868190+0000 mon.b (mon.2) 9 : audit 1 from='osd.7 [v2:172.21.15.169:6824/1292684097,v1:172.21.15.169:6825/1292684097]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-12-16T07:51:41.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:41 smithi007 bash[65864]: audit 2023-12-16T07:51:40.516994+0000 mon.a (mon.0) 567 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:41.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:41 smithi007 bash[65864]: audit 2023-12-16T07:51:40.527587+0000 mon.a (mon.0) 568 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:41.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:41 smithi007 bash[65864]: audit 2023-12-16T07:51:40.866570+0000 mon.a (mon.0) 569 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-12-16T07:51:41.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:41 smithi007 bash[65864]: audit 2023-12-16T07:51:40.868190+0000 mon.b (mon.2) 9 : audit 1 from='osd.7 [v2:172.21.15.169:6824/1292684097,v1:172.21.15.169:6825/1292684097]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-12-16T07:51:41.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:41 smithi169 bash[51638]: audit 2023-12-16T07:51:40.516994+0000 mon.a (mon.0) 567 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:41.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:41 smithi169 bash[51638]: audit 2023-12-16T07:51:40.527587+0000 mon.a (mon.0) 568 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:41.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:41 smithi169 bash[51638]: audit 2023-12-16T07:51:40.866570+0000 mon.a (mon.0) 569 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-12-16T07:51:41.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:41 smithi169 bash[51638]: audit 2023-12-16T07:51:40.868190+0000 mon.b (mon.2) 9 : audit 1 from='osd.7 [v2:172.21.15.169:6824/1292684097,v1:172.21.15.169:6825/1292684097]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-12-16T07:51:41.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:51:41 smithi169 bash[60136]: debug 2023-12-16T07:51:41.602+0000 7f5060ec6700 -1 osd.7 138 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-16T07:51:42.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:42 smithi007 bash[64479]: cluster 2023-12-16T07:51:41.293090+0000 mgr.y (mgr.25477) 340 : cluster 0 pgmap v214: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:51:42.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:42 smithi007 bash[64479]: audit 2023-12-16T07:51:41.531556+0000 mon.a (mon.0) 570 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-12-16T07:51:42.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:42 smithi007 bash[64479]: cluster 2023-12-16T07:51:41.535985+0000 mon.a (mon.0) 571 : cluster 0 osdmap e141: 8 total, 7 up, 8 in 2023-12-16T07:51:42.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:42 smithi007 bash[64479]: audit 2023-12-16T07:51:41.540954+0000 mon.a (mon.0) 572 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:51:42.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:42 smithi007 bash[64479]: audit 2023-12-16T07:51:41.542805+0000 mon.b (mon.2) 10 : audit 1 from='osd.7 [v2:172.21.15.169:6824/1292684097,v1:172.21.15.169:6825/1292684097]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:51:42.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:42 smithi007 bash[64479]: audit 2023-12-16T07:51:41.625017+0000 mon.a (mon.0) 573 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:42.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:42 smithi007 bash[64479]: audit 2023-12-16T07:51:41.637489+0000 mon.a (mon.0) 574 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:42.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:42 smithi007 bash[65864]: cluster 2023-12-16T07:51:41.293090+0000 mgr.y (mgr.25477) 340 : cluster 0 pgmap v214: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:51:42.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:42 smithi007 bash[65864]: audit 2023-12-16T07:51:41.531556+0000 mon.a (mon.0) 570 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-12-16T07:51:42.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:42 smithi007 bash[65864]: cluster 2023-12-16T07:51:41.535985+0000 mon.a (mon.0) 571 : cluster 0 osdmap e141: 8 total, 7 up, 8 in 2023-12-16T07:51:42.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:42 smithi007 bash[65864]: audit 2023-12-16T07:51:41.540954+0000 mon.a (mon.0) 572 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:51:42.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:42 smithi007 bash[65864]: audit 2023-12-16T07:51:41.542805+0000 mon.b (mon.2) 10 : audit 1 from='osd.7 [v2:172.21.15.169:6824/1292684097,v1:172.21.15.169:6825/1292684097]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:51:42.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:42 smithi007 bash[65864]: audit 2023-12-16T07:51:41.625017+0000 mon.a (mon.0) 573 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:42.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:42 smithi007 bash[65864]: audit 2023-12-16T07:51:41.637489+0000 mon.a (mon.0) 574 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:42.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:42 smithi169 bash[51638]: cluster 2023-12-16T07:51:41.293090+0000 mgr.y (mgr.25477) 340 : cluster 0 pgmap v214: 161 pgs: 43 active+undersized, 26 active+undersized+degraded, 92 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 82/630 objects degraded (13.016%) 2023-12-16T07:51:42.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:42 smithi169 bash[51638]: audit 2023-12-16T07:51:41.531556+0000 mon.a (mon.0) 570 : audit 1 from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-12-16T07:51:42.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:42 smithi169 bash[51638]: cluster 2023-12-16T07:51:41.535985+0000 mon.a (mon.0) 571 : cluster 0 osdmap e141: 8 total, 7 up, 8 in 2023-12-16T07:51:42.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:42 smithi169 bash[51638]: audit 2023-12-16T07:51:41.540954+0000 mon.a (mon.0) 572 : audit 1 from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:51:42.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:42 smithi169 bash[51638]: audit 2023-12-16T07:51:41.542805+0000 mon.b (mon.2) 10 : audit 1 from='osd.7 [v2:172.21.15.169:6824/1292684097,v1:172.21.15.169:6825/1292684097]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi169", "root=default"]}]: dispatch 2023-12-16T07:51:42.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:42 smithi169 bash[51638]: audit 2023-12-16T07:51:41.625017+0000 mon.a (mon.0) 573 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:42.934 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:42 smithi169 bash[51638]: audit 2023-12-16T07:51:41.637489+0000 mon.a (mon.0) 574 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:43.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:43 smithi007 bash[65864]: cluster 2023-12-16T07:51:42.532547+0000 mon.a (mon.0) 575 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:51:43.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:43 smithi007 bash[65864]: cluster 2023-12-16T07:51:42.543144+0000 mon.a (mon.0) 576 : cluster 1 osd.7 [v2:172.21.15.169:6824/1292684097,v1:172.21.15.169:6825/1292684097] boot 2023-12-16T07:51:43.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:43 smithi007 bash[65864]: cluster 2023-12-16T07:51:42.543337+0000 mon.a (mon.0) 577 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2023-12-16T07:51:43.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:43 smithi007 bash[65864]: audit 2023-12-16T07:51:42.547866+0000 mon.a (mon.0) 578 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:51:43.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:43 smithi007 bash[65864]: cluster 2023-12-16T07:51:43.544885+0000 mon.a (mon.0) 579 : cluster 0 osdmap e143: 8 total, 8 up, 8 in 2023-12-16T07:51:43.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:43 smithi007 bash[64479]: cluster 2023-12-16T07:51:42.532547+0000 mon.a (mon.0) 575 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:51:43.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:43 smithi007 bash[64479]: cluster 2023-12-16T07:51:42.543144+0000 mon.a (mon.0) 576 : cluster 1 osd.7 [v2:172.21.15.169:6824/1292684097,v1:172.21.15.169:6825/1292684097] boot 2023-12-16T07:51:43.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:43 smithi007 bash[64479]: cluster 2023-12-16T07:51:42.543337+0000 mon.a (mon.0) 577 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2023-12-16T07:51:43.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:43 smithi007 bash[64479]: audit 2023-12-16T07:51:42.547866+0000 mon.a (mon.0) 578 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:51:43.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:43 smithi007 bash[64479]: cluster 2023-12-16T07:51:43.544885+0000 mon.a (mon.0) 579 : cluster 0 osdmap e143: 8 total, 8 up, 8 in 2023-12-16T07:51:43.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:43 smithi169 bash[51638]: cluster 2023-12-16T07:51:42.532547+0000 mon.a (mon.0) 575 : cluster 1 Health check cleared: OSD_DOWN (was: 1 osds down) 2023-12-16T07:51:43.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:43 smithi169 bash[51638]: cluster 2023-12-16T07:51:42.543144+0000 mon.a (mon.0) 576 : cluster 1 osd.7 [v2:172.21.15.169:6824/1292684097,v1:172.21.15.169:6825/1292684097] boot 2023-12-16T07:51:43.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:43 smithi169 bash[51638]: cluster 2023-12-16T07:51:42.543337+0000 mon.a (mon.0) 577 : cluster 0 osdmap e142: 8 total, 8 up, 8 in 2023-12-16T07:51:43.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:43 smithi169 bash[51638]: audit 2023-12-16T07:51:42.547866+0000 mon.a (mon.0) 578 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-12-16T07:51:43.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:43 smithi169 bash[51638]: cluster 2023-12-16T07:51:43.544885+0000 mon.a (mon.0) 579 : cluster 0 osdmap e143: 8 total, 8 up, 8 in 2023-12-16T07:51:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:44 smithi007 bash[65864]: cluster 2023-12-16T07:51:43.294426+0000 mgr.y (mgr.25477) 341 : cluster 0 pgmap v217: 161 pgs: 10 peering, 37 active+undersized, 22 active+undersized+degraded, 92 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 69/630 objects degraded (10.952%) 2023-12-16T07:51:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:44 smithi007 bash[65864]: audit 2023-12-16T07:51:43.580494+0000 mon.a (mon.0) 580 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:44.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:44 smithi007 bash[65864]: cluster 2023-12-16T07:51:43.886226+0000 mon.a (mon.0) 581 : cluster 3 Health check update: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:44 smithi007 bash[64479]: cluster 2023-12-16T07:51:43.294426+0000 mgr.y (mgr.25477) 341 : cluster 0 pgmap v217: 161 pgs: 10 peering, 37 active+undersized, 22 active+undersized+degraded, 92 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 69/630 objects degraded (10.952%) 2023-12-16T07:51:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:44 smithi007 bash[64479]: audit 2023-12-16T07:51:43.580494+0000 mon.a (mon.0) 580 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:44.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:44 smithi007 bash[64479]: cluster 2023-12-16T07:51:43.886226+0000 mon.a (mon.0) 581 : cluster 3 Health check update: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:44 smithi169 bash[51638]: cluster 2023-12-16T07:51:43.294426+0000 mgr.y (mgr.25477) 341 : cluster 0 pgmap v217: 161 pgs: 10 peering, 37 active+undersized, 22 active+undersized+degraded, 92 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 69/630 objects degraded (10.952%) 2023-12-16T07:51:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:44 smithi169 bash[51638]: audit 2023-12-16T07:51:43.580494+0000 mon.a (mon.0) 580 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:44.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:44 smithi169 bash[51638]: cluster 2023-12-16T07:51:43.886226+0000 mon.a (mon.0) 581 : cluster 3 Health check update: Degraded data redundancy: 69/630 objects degraded (10.952%), 22 pgs degraded (PG_DEGRADED) 2023-12-16T07:51:45.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:45 smithi169 bash[51638]: audit 2023-12-16T07:51:44.389794+0000 mgr.y (mgr.25477) 342 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:46.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:45 smithi007 bash[64479]: audit 2023-12-16T07:51:44.389794+0000 mgr.y (mgr.25477) 342 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:46.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:45 smithi007 bash[65864]: audit 2023-12-16T07:51:44.389794+0000 mgr.y (mgr.25477) 342 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:46.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:46 smithi169 bash[51638]: cluster 2023-12-16T07:51:45.295897+0000 mgr.y (mgr.25477) 343 : cluster 0 pgmap v219: 161 pgs: 21 peering, 17 active+undersized, 9 active+undersized+degraded, 114 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 38/630 objects degraded (6.032%) 2023-12-16T07:51:47.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:46 smithi007 bash[64479]: cluster 2023-12-16T07:51:45.295897+0000 mgr.y (mgr.25477) 343 : cluster 0 pgmap v219: 161 pgs: 21 peering, 17 active+undersized, 9 active+undersized+degraded, 114 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 38/630 objects degraded (6.032%) 2023-12-16T07:51:47.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:46 smithi007 bash[65864]: cluster 2023-12-16T07:51:45.295897+0000 mgr.y (mgr.25477) 343 : cluster 0 pgmap v219: 161 pgs: 21 peering, 17 active+undersized, 9 active+undersized+degraded, 114 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 38/630 objects degraded (6.032%) 2023-12-16T07:51:47.914 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:47 smithi169 bash[51638]: cluster 2023-12-16T07:51:47.297020+0000 mgr.y (mgr.25477) 344 : cluster 0 pgmap v220: 161 pgs: 21 peering, 140 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:51:48.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:47 smithi007 bash[65864]: cluster 2023-12-16T07:51:47.297020+0000 mgr.y (mgr.25477) 344 : cluster 0 pgmap v220: 161 pgs: 21 peering, 140 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:51:48.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:47 smithi007 bash[64479]: cluster 2023-12-16T07:51:47.297020+0000 mgr.y (mgr.25477) 344 : cluster 0 pgmap v220: 161 pgs: 21 peering, 140 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2023-12-16T07:51:48.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:48 smithi169 bash[51638]: cluster 2023-12-16T07:51:47.578027+0000 mon.a (mon.0) 582 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 9 pgs degraded) 2023-12-16T07:51:48.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:48 smithi169 bash[51638]: cluster 2023-12-16T07:51:47.578093+0000 mon.a (mon.0) 583 : cluster 1 Cluster is now healthy 2023-12-16T07:51:49.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:48 smithi007 bash[64479]: cluster 2023-12-16T07:51:47.578027+0000 mon.a (mon.0) 582 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 9 pgs degraded) 2023-12-16T07:51:49.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:48 smithi007 bash[64479]: cluster 2023-12-16T07:51:47.578093+0000 mon.a (mon.0) 583 : cluster 1 Cluster is now healthy 2023-12-16T07:51:49.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:48 smithi007 bash[65864]: cluster 2023-12-16T07:51:47.578027+0000 mon.a (mon.0) 582 : cluster 1 Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 38/630 objects degraded (6.032%), 9 pgs degraded) 2023-12-16T07:51:49.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:48 smithi007 bash[65864]: cluster 2023-12-16T07:51:47.578093+0000 mon.a (mon.0) 583 : cluster 1 Cluster is now healthy 2023-12-16T07:51:50.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:49 smithi169 bash[51638]: cluster 2023-12-16T07:51:49.298387+0000 mgr.y (mgr.25477) 345 : cluster 0 pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2023-12-16T07:51:50.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:49 smithi007 bash[65864]: cluster 2023-12-16T07:51:49.298387+0000 mgr.y (mgr.25477) 345 : cluster 0 pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2023-12-16T07:51:50.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:49 smithi007 bash[64479]: cluster 2023-12-16T07:51:49.298387+0000 mgr.y (mgr.25477) 345 : cluster 0 pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 791 B/s rd, 0 op/s 2023-12-16T07:51:50.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:51:50 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:51:50] "GET /metrics HTTP/1.1" 200 38438 "" "Prometheus/2.33.4" 2023-12-16T07:51:52.318 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:51:51 smithi007 bash[61260]: debug 2023-12-16T07:51:51.841+0000 7f2cb18fa700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception 2023-12-16T07:51:52.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: cluster 2023-12-16T07:51:51.299649+0000 mgr.y (mgr.25477) 346 : cluster 0 pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:51:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.637950+0000 mon.a (mon.0) 584 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.648659+0000 mon.a (mon.0) 585 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.651283+0000 mon.a (mon.0) 586 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:51:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.653289+0000 mon.a (mon.0) 587 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:51:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.662526+0000 mon.a (mon.0) 588 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.751043+0000 mon.a (mon.0) 589 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:51:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.754217+0000 mon.a (mon.0) 590 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.758303+0000 mon.a (mon.0) 591 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.760887+0000 mon.a (mon.0) 592 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.766381+0000 mon.a (mon.0) 593 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.774901+0000 mon.a (mon.0) 594 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.779180+0000 mon.a (mon.0) 595 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2023-12-16T07:51:52.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.785411+0000 mon.a (mon.0) 596 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2023-12-16T07:51:52.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.789201+0000 mon.a (mon.0) 597 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2023-12-16T07:51:52.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.793670+0000 mon.a (mon.0) 598 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2023-12-16T07:51:52.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.796695+0000 mon.a (mon.0) 599 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2023-12-16T07:51:52.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.800971+0000 mon.a (mon.0) 600 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2023-12-16T07:51:52.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.804136+0000 mon.a (mon.0) 601 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2023-12-16T07:51:52.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.808396+0000 mon.a (mon.0) 602 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2023-12-16T07:51:52.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.811530+0000 mon.a (mon.0) 603 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2023-12-16T07:51:52.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.816072+0000 mon.a (mon.0) 604 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2023-12-16T07:51:52.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.819319+0000 mon.a (mon.0) 605 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2023-12-16T07:51:52.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.823479+0000 mon.a (mon.0) 606 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2023-12-16T07:51:52.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.826389+0000 mon.a (mon.0) 607 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2023-12-16T07:51:52.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.830886+0000 mon.a (mon.0) 608 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2023-12-16T07:51:52.685 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.833830+0000 mon.a (mon.0) 609 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2023-12-16T07:51:52.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.838410+0000 mon.a (mon.0) 610 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2023-12-16T07:51:52.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.843876+0000 mon.a (mon.0) 611 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "unknown"}]: dispatch 2023-12-16T07:51:52.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.855624+0000 mon.a (mon.0) 612 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.859167+0000 mon.a (mon.0) 613 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:51:52.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.860555+0000 mon.a (mon.0) 614 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:51:52.686 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:52 smithi169 bash[51638]: audit 2023-12-16T07:51:51.868374+0000 mon.a (mon.0) 615 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: cluster 2023-12-16T07:51:51.299649+0000 mgr.y (mgr.25477) 346 : cluster 0 pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:51:52.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.637950+0000 mon.a (mon.0) 584 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.648659+0000 mon.a (mon.0) 585 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.651283+0000 mon.a (mon.0) 586 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.653289+0000 mon.a (mon.0) 587 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.662526+0000 mon.a (mon.0) 588 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.751043+0000 mon.a (mon.0) 589 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.754217+0000 mon.a (mon.0) 590 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.758303+0000 mon.a (mon.0) 591 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.760887+0000 mon.a (mon.0) 592 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.766381+0000 mon.a (mon.0) 593 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.774901+0000 mon.a (mon.0) 594 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.779180+0000 mon.a (mon.0) 595 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.785411+0000 mon.a (mon.0) 596 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.789201+0000 mon.a (mon.0) 597 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.793670+0000 mon.a (mon.0) 598 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.796695+0000 mon.a (mon.0) 599 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2023-12-16T07:51:52.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.800971+0000 mon.a (mon.0) 600 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2023-12-16T07:51:52.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.804136+0000 mon.a (mon.0) 601 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2023-12-16T07:51:52.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: cluster 2023-12-16T07:51:51.299649+0000 mgr.y (mgr.25477) 346 : cluster 0 pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:51:52.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.637950+0000 mon.a (mon.0) 584 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.648659+0000 mon.a (mon.0) 585 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.651283+0000 mon.a (mon.0) 586 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:51:52.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.653289+0000 mon.a (mon.0) 587 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:51:52.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.662526+0000 mon.a (mon.0) 588 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.751043+0000 mon.a (mon.0) 589 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-16T07:51:52.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.754217+0000 mon.a (mon.0) 590 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:52.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.758303+0000 mon.a (mon.0) 591 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:52.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.760887+0000 mon.a (mon.0) 592 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:52.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.766381+0000 mon.a (mon.0) 593 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:51:52.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.774901+0000 mon.a (mon.0) 594 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.779180+0000 mon.a (mon.0) 595 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2023-12-16T07:51:52.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.785411+0000 mon.a (mon.0) 596 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2023-12-16T07:51:52.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.789201+0000 mon.a (mon.0) 597 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2023-12-16T07:51:52.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.793670+0000 mon.a (mon.0) 598 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2023-12-16T07:51:52.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.796695+0000 mon.a (mon.0) 599 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2023-12-16T07:51:52.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.800971+0000 mon.a (mon.0) 600 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2023-12-16T07:51:52.821 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.804136+0000 mon.a (mon.0) 601 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2023-12-16T07:51:52.821 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.808396+0000 mon.a (mon.0) 602 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2023-12-16T07:51:52.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.811530+0000 mon.a (mon.0) 603 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2023-12-16T07:51:52.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.816072+0000 mon.a (mon.0) 604 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2023-12-16T07:51:52.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.819319+0000 mon.a (mon.0) 605 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2023-12-16T07:51:52.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.823479+0000 mon.a (mon.0) 606 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2023-12-16T07:51:52.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.826389+0000 mon.a (mon.0) 607 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2023-12-16T07:51:52.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.830886+0000 mon.a (mon.0) 608 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2023-12-16T07:51:52.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.833830+0000 mon.a (mon.0) 609 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2023-12-16T07:51:52.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.838410+0000 mon.a (mon.0) 610 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2023-12-16T07:51:52.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.843876+0000 mon.a (mon.0) 611 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "unknown"}]: dispatch 2023-12-16T07:51:52.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.855624+0000 mon.a (mon.0) 612 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.859167+0000 mon.a (mon.0) 613 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:51:52.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.860555+0000 mon.a (mon.0) 614 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:51:52.822 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: audit 2023-12-16T07:51:51.868374+0000 mon.a (mon.0) 615 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.808396+0000 mon.a (mon.0) 602 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2023-12-16T07:51:52.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.811530+0000 mon.a (mon.0) 603 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2023-12-16T07:51:52.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.816072+0000 mon.a (mon.0) 604 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2023-12-16T07:51:52.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.819319+0000 mon.a (mon.0) 605 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2023-12-16T07:51:52.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.823479+0000 mon.a (mon.0) 606 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2023-12-16T07:51:52.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.826389+0000 mon.a (mon.0) 607 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2023-12-16T07:51:52.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.830886+0000 mon.a (mon.0) 608 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2023-12-16T07:51:52.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.833830+0000 mon.a (mon.0) 609 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2023-12-16T07:51:52.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.838410+0000 mon.a (mon.0) 610 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2023-12-16T07:51:52.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.843876+0000 mon.a (mon.0) 611 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "unknown"}]: dispatch 2023-12-16T07:51:52.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.855624+0000 mon.a (mon.0) 612 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:52.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.859167+0000 mon.a (mon.0) 613 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:51:52.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.860555+0000 mon.a (mon.0) 614 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:51:52.823 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[65864]: audit 2023-12-16T07:51:51.868374+0000 mon.a (mon.0) 615 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:52 smithi007 bash[64479]: debug 2023-12-16T07:51:52.849+0000 7f4f5c70b700 -1 log_channel(cluster) log [ERR] : Health check failed: Upgrade: failed due to an unexpected exception (UPGRADE_EXCEPTION) 2023-12-16T07:51:53.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:53 smithi169 bash[51638]: cephadm 2023-12-16T07:51:51.767840+0000 mgr.y (mgr.25477) 347 : cephadm 1 Upgrade: Setting container_image for all osd 2023-12-16T07:51:53.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:53 smithi169 bash[51638]: cephadm 2023-12-16T07:51:51.843379+0000 mgr.y (mgr.25477) 348 : cephadm 1 Upgrade: Setting require_osd_release to 19 unknown 2023-12-16T07:51:53.683 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:53 smithi169 bash[51638]: cephadm 2023-12-16T07:51:51.845019+0000 mgr.y (mgr.25477) 349 : cephadm 4 Upgrade: Paused due to UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception 2023-12-16T07:51:53.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:53 smithi169 bash[51638]: cluster 2023-12-16T07:51:51.857405+0000 mgr.y (mgr.25477) 350 : cluster 0 pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:51:53.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:53 smithi169 bash[51638]: cluster 2023-12-16T07:51:52.852288+0000 mon.a (mon.0) 616 : cluster 4 Health check failed: Upgrade: failed due to an unexpected exception (UPGRADE_EXCEPTION) 2023-12-16T07:51:53.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:53 smithi169 bash[51638]: audit 2023-12-16T07:51:53.214399+0000 mon.a (mon.0) 617 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:53.684 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:53 smithi169 bash[51638]: audit 2023-12-16T07:51:53.216480+0000 mon.a (mon.0) 618 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:51:53.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:53 smithi007 bash[64479]: cephadm 2023-12-16T07:51:51.767840+0000 mgr.y (mgr.25477) 347 : cephadm 1 Upgrade: Setting container_image for all osd 2023-12-16T07:51:53.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:53 smithi007 bash[64479]: cephadm 2023-12-16T07:51:51.843379+0000 mgr.y (mgr.25477) 348 : cephadm 1 Upgrade: Setting require_osd_release to 19 unknown 2023-12-16T07:51:53.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:53 smithi007 bash[64479]: cephadm 2023-12-16T07:51:51.845019+0000 mgr.y (mgr.25477) 349 : cephadm 4 Upgrade: Paused due to UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception 2023-12-16T07:51:53.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:53 smithi007 bash[64479]: cluster 2023-12-16T07:51:51.857405+0000 mgr.y (mgr.25477) 350 : cluster 0 pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:51:53.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:53 smithi007 bash[64479]: cluster 2023-12-16T07:51:52.852288+0000 mon.a (mon.0) 616 : cluster 4 Health check failed: Upgrade: failed due to an unexpected exception (UPGRADE_EXCEPTION) 2023-12-16T07:51:53.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:53 smithi007 bash[64479]: audit 2023-12-16T07:51:53.214399+0000 mon.a (mon.0) 617 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:53.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:53 smithi007 bash[64479]: audit 2023-12-16T07:51:53.216480+0000 mon.a (mon.0) 618 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:51:53.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:53 smithi007 bash[65864]: cephadm 2023-12-16T07:51:51.767840+0000 mgr.y (mgr.25477) 347 : cephadm 1 Upgrade: Setting container_image for all osd 2023-12-16T07:51:53.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:53 smithi007 bash[65864]: cephadm 2023-12-16T07:51:51.843379+0000 mgr.y (mgr.25477) 348 : cephadm 1 Upgrade: Setting require_osd_release to 19 unknown 2023-12-16T07:51:53.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:53 smithi007 bash[65864]: cephadm 2023-12-16T07:51:51.845019+0000 mgr.y (mgr.25477) 349 : cephadm 4 Upgrade: Paused due to UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception 2023-12-16T07:51:53.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:53 smithi007 bash[65864]: cluster 2023-12-16T07:51:51.857405+0000 mgr.y (mgr.25477) 350 : cluster 0 pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2023-12-16T07:51:53.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:53 smithi007 bash[65864]: cluster 2023-12-16T07:51:52.852288+0000 mon.a (mon.0) 616 : cluster 4 Health check failed: Upgrade: failed due to an unexpected exception (UPGRADE_EXCEPTION) 2023-12-16T07:51:53.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:53 smithi007 bash[65864]: audit 2023-12-16T07:51:53.214399+0000 mon.a (mon.0) 617 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:53.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:53 smithi007 bash[65864]: audit 2023-12-16T07:51:53.216480+0000 mon.a (mon.0) 618 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:51:54.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:54 smithi007 bash[64479]: audit 2023-12-16T07:51:53.610329+0000 mon.a (mon.0) 619 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:54.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:54 smithi007 bash[64479]: cluster 2023-12-16T07:51:53.858674+0000 mgr.y (mgr.25477) 351 : cluster 0 pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:51:54.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:54 smithi007 bash[64479]: audit 2023-12-16T07:51:54.392370+0000 mgr.y (mgr.25477) 352 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:54.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:54 smithi007 bash[65864]: audit 2023-12-16T07:51:53.610329+0000 mon.a (mon.0) 619 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:54.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:54 smithi007 bash[65864]: cluster 2023-12-16T07:51:53.858674+0000 mgr.y (mgr.25477) 351 : cluster 0 pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:51:54.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:54 smithi007 bash[65864]: audit 2023-12-16T07:51:54.392370+0000 mgr.y (mgr.25477) 352 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:54.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:54 smithi169 bash[51638]: audit 2023-12-16T07:51:53.610329+0000 mon.a (mon.0) 619 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:51:54.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:54 smithi169 bash[51638]: cluster 2023-12-16T07:51:53.858674+0000 mgr.y (mgr.25477) 351 : cluster 0 pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2023-12-16T07:51:54.933 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:54 smithi169 bash[51638]: audit 2023-12-16T07:51:54.392370+0000 mgr.y (mgr.25477) 352 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:51:56.084 INFO:teuthology.orchestra.run.smithi007.stdout:true 2023-12-16T07:51:56.454 INFO:teuthology.orchestra.run.smithi007.stdout:"Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception" 2023-12-16T07:51:57.192 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:56 smithi169 bash[51638]: cluster 2023-12-16T07:51:55.859763+0000 mgr.y (mgr.25477) 353 : cluster 0 pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-12-16T07:51:57.192 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:56 smithi169 bash[51638]: audit 2023-12-16T07:51:56.066201+0000 mgr.y (mgr.25477) 354 : audit 0 from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:57.192 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:56 smithi169 bash[51638]: audit 2023-12-16T07:51:56.436566+0000 mgr.y (mgr.25477) 355 : audit 0 from='client.44506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:57.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:56 smithi007 bash[64479]: cluster 2023-12-16T07:51:55.859763+0000 mgr.y (mgr.25477) 353 : cluster 0 pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-12-16T07:51:57.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:56 smithi007 bash[64479]: audit 2023-12-16T07:51:56.066201+0000 mgr.y (mgr.25477) 354 : audit 0 from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:57.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:56 smithi007 bash[64479]: audit 2023-12-16T07:51:56.436566+0000 mgr.y (mgr.25477) 355 : audit 0 from='client.44506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:57.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:56 smithi007 bash[65864]: cluster 2023-12-16T07:51:55.859763+0000 mgr.y (mgr.25477) 353 : cluster 0 pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2023-12-16T07:51:57.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:56 smithi007 bash[65864]: audit 2023-12-16T07:51:56.066201+0000 mgr.y (mgr.25477) 354 : audit 0 from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:57.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:56 smithi007 bash[65864]: audit 2023-12-16T07:51:56.436566+0000 mgr.y (mgr.25477) 355 : audit 0 from='client.44506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:51:57.344 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph orch ps' 2023-12-16T07:51:59.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:51:58 smithi169 bash[51638]: cluster 2023-12-16T07:51:57.861139+0000 mgr.y (mgr.25477) 356 : cluster 0 pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2023-12-16T07:51:59.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:51:58 smithi007 bash[64479]: cluster 2023-12-16T07:51:57.861139+0000 mgr.y (mgr.25477) 356 : cluster 0 pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2023-12-16T07:51:59.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:51:58 smithi007 bash[65864]: cluster 2023-12-16T07:51:57.861139+0000 mgr.y (mgr.25477) 356 : cluster 0 pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2023-12-16T07:52:00.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:52:00 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:52:00] "GET /metrics HTTP/1.1" 200 38509 "" "Prometheus/2.33.4" 2023-12-16T07:52:00.700 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:52:00.700 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (8m) 2m ago 15m 16.8M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:52:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (7m) 20s ago 15m 44.7M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:52:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (6m) 2m ago 13m 47.7M - 3.5 e1d6a67b021e 8d8009d0c83f 2023-12-16T07:52:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283,8765 running (6m) 20s ago 21m 436M - 19.0.0-191-g6d933094 a29c258b5d6f 5295ba35c2cd 2023-12-16T07:52:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (7m) 2m ago 23m 493M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:52:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (6m) 2m ago 23m 52.7M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 5a266b45f3ce 2023-12-16T07:52:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (5m) 20s ago 21m 45.5M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1fc9bf864dda 2023-12-16T07:52:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (5m) 2m ago 22m 42.0M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1dc91c31fc27 2023-12-16T07:52:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (8m) 2m ago 15m 13.0M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:52:00.701 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (8m) 20s ago 15m 15.2M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:52:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (3m) 2m ago 21m 52.0M 2782M 19.0.0-191-g6d933094 a29c258b5d6f a00724c6d077 2023-12-16T07:52:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (3m) 2m ago 20m 51.3M 2782M 19.0.0-191-g6d933094 a29c258b5d6f 5e90fce88079 2023-12-16T07:52:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (2m) 2m ago 19m 46.9M 2782M 19.0.0-191-g6d933094 a29c258b5d6f 906a30291e88 2023-12-16T07:52:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (2m) 2m ago 19m - 2782M 19.0.0-191-g6d933094 a29c258b5d6f 8e5e7d6ede0f 2023-12-16T07:52:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (112s) 20s ago 18m 52.9M 3037M 19.0.0-191-g6d933094 a29c258b5d6f b612e0176579 2023-12-16T07:52:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (82s) 20s ago 18m 49.6M 3037M 19.0.0-191-g6d933094 a29c258b5d6f c1ca50cfe550 2023-12-16T07:52:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (53s) 20s ago 17m 48.2M 3037M 19.0.0-191-g6d933094 a29c258b5d6f b335dc393bbe 2023-12-16T07:52:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (24s) 20s ago 16m - 3037M 19.0.0-191-g6d933094 a29c258b5d6f 0914497c21be 2023-12-16T07:52:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (6m) 20s ago 15m 64.6M - 2.33.4 514e6a882f6e 2652ce1b8540 2023-12-16T07:52:00.702 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (13m) 2m ago 13m 88.1M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:52:00.703 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (13m) 20s ago 13m 88.9M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:52:00.703 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (13m) 2m ago 13m 88.0M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:52:00.703 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (13m) 20s ago 13m 88.7M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:52:01.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:00 smithi007 bash[64479]: cluster 2023-12-16T07:51:59.862711+0000 mgr.y (mgr.25477) 357 : cluster 0 pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2023-12-16T07:52:01.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:00 smithi007 bash[65864]: cluster 2023-12-16T07:51:59.862711+0000 mgr.y (mgr.25477) 357 : cluster 0 pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2023-12-16T07:52:01.183 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:00 smithi169 bash[51638]: cluster 2023-12-16T07:51:59.862711+0000 mgr.y (mgr.25477) 357 : cluster 0 pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2023-12-16T07:52:01.438 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph versions' 2023-12-16T07:52:02.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:01 smithi007 bash[64479]: audit 2023-12-16T07:52:00.672233+0000 mgr.y (mgr.25477) 358 : audit 0 from='client.34581 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:52:02.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:01 smithi007 bash[65864]: audit 2023-12-16T07:52:00.672233+0000 mgr.y (mgr.25477) 358 : audit 0 from='client.34581 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:52:02.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:01 smithi169 bash[51638]: audit 2023-12-16T07:52:00.672233+0000 mgr.y (mgr.25477) 358 : audit 0 from='client.34581 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:52:03.068 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:02 smithi007 bash[64479]: cluster 2023-12-16T07:52:01.863599+0000 mgr.y (mgr.25477) 359 : cluster 0 pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2023-12-16T07:52:03.068 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:02 smithi007 bash[65864]: cluster 2023-12-16T07:52:01.863599+0000 mgr.y (mgr.25477) 359 : cluster 0 pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2023-12-16T07:52:03.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:02 smithi169 bash[51638]: cluster 2023-12-16T07:52:01.863599+0000 mgr.y (mgr.25477) 359 : cluster 0 pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2023-12-16T07:52:04.831 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:52:04.831 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 3 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 8 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 13 2023-12-16T07:52:04.832 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:52:04.833 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:52:05.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:04 smithi007 bash[64479]: cluster 2023-12-16T07:52:03.864857+0000 mgr.y (mgr.25477) 360 : cluster 0 pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:05.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:04 smithi007 bash[64479]: audit 2023-12-16T07:52:04.403142+0000 mgr.y (mgr.25477) 361 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:05.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:04 smithi007 bash[64479]: audit 2023-12-16T07:52:04.828265+0000 mon.a (mon.0) 620 : audit 0 from='client.? 172.21.15.7:0/761679379' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:52:05.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:04 smithi007 bash[65864]: cluster 2023-12-16T07:52:03.864857+0000 mgr.y (mgr.25477) 360 : cluster 0 pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:05.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:04 smithi007 bash[65864]: audit 2023-12-16T07:52:04.403142+0000 mgr.y (mgr.25477) 361 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:05.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:04 smithi007 bash[65864]: audit 2023-12-16T07:52:04.828265+0000 mon.a (mon.0) 620 : audit 0 from='client.? 172.21.15.7:0/761679379' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:52:05.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:04 smithi169 bash[51638]: cluster 2023-12-16T07:52:03.864857+0000 mgr.y (mgr.25477) 360 : cluster 0 pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:05.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:04 smithi169 bash[51638]: audit 2023-12-16T07:52:04.403142+0000 mgr.y (mgr.25477) 361 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:05.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:04 smithi169 bash[51638]: audit 2023-12-16T07:52:04.828265+0000 mon.a (mon.0) 620 : audit 0 from='client.? 172.21.15.7:0/761679379' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:52:05.577 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2023-12-16T07:52:07.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:06 smithi007 bash[65864]: cluster 2023-12-16T07:52:05.866156+0000 mgr.y (mgr.25477) 362 : cluster 0 pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:07.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:06 smithi007 bash[64479]: cluster 2023-12-16T07:52:05.866156+0000 mgr.y (mgr.25477) 362 : cluster 0 pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:07.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:06 smithi169 bash[51638]: cluster 2023-12-16T07:52:05.866156+0000 mgr.y (mgr.25477) 362 : cluster 0 pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:08.383 INFO:teuthology.orchestra.run.smithi007.stdout:wait for servicemap items w/ changing names to refresh 2023-12-16T07:52:09.127 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'sleep 60' 2023-12-16T07:52:09.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:08 smithi007 bash[64479]: cluster 2023-12-16T07:52:07.867473+0000 mgr.y (mgr.25477) 363 : cluster 0 pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:09.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:08 smithi007 bash[64479]: audit 2023-12-16T07:52:08.208734+0000 mon.a (mon.0) 621 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:52:09.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:08 smithi007 bash[65864]: cluster 2023-12-16T07:52:07.867473+0000 mgr.y (mgr.25477) 363 : cluster 0 pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:09.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:08 smithi007 bash[65864]: audit 2023-12-16T07:52:08.208734+0000 mon.a (mon.0) 621 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:52:09.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:08 smithi169 bash[51638]: cluster 2023-12-16T07:52:07.867473+0000 mgr.y (mgr.25477) 363 : cluster 0 pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:09.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:08 smithi169 bash[51638]: audit 2023-12-16T07:52:08.208734+0000 mon.a (mon.0) 621 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:52:10.493 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:52:10 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:52:10] "GET /metrics HTTP/1.1" 200 38511 "" "Prometheus/2.33.4" 2023-12-16T07:52:11.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:10 smithi007 bash[64479]: cluster 2023-12-16T07:52:09.869043+0000 mgr.y (mgr.25477) 364 : cluster 0 pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:11.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:10 smithi007 bash[65864]: cluster 2023-12-16T07:52:09.869043+0000 mgr.y (mgr.25477) 364 : cluster 0 pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:11.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:10 smithi169 bash[51638]: cluster 2023-12-16T07:52:09.869043+0000 mgr.y (mgr.25477) 364 : cluster 0 pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:13.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:12 smithi007 bash[64479]: cluster 2023-12-16T07:52:11.869973+0000 mgr.y (mgr.25477) 365 : cluster 0 pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:13.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:12 smithi007 bash[65864]: cluster 2023-12-16T07:52:11.869973+0000 mgr.y (mgr.25477) 365 : cluster 0 pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:13.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:12 smithi169 bash[51638]: cluster 2023-12-16T07:52:11.869973+0000 mgr.y (mgr.25477) 365 : cluster 0 pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:15.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:14 smithi007 bash[64479]: cluster 2023-12-16T07:52:13.871312+0000 mgr.y (mgr.25477) 366 : cluster 0 pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:15.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:14 smithi007 bash[64479]: audit 2023-12-16T07:52:14.407205+0000 mgr.y (mgr.25477) 367 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:15.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:14 smithi007 bash[65864]: cluster 2023-12-16T07:52:13.871312+0000 mgr.y (mgr.25477) 366 : cluster 0 pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:15.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:14 smithi007 bash[65864]: audit 2023-12-16T07:52:14.407205+0000 mgr.y (mgr.25477) 367 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:15.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:14 smithi169 bash[51638]: cluster 2023-12-16T07:52:13.871312+0000 mgr.y (mgr.25477) 366 : cluster 0 pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:15.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:14 smithi169 bash[51638]: audit 2023-12-16T07:52:14.407205+0000 mgr.y (mgr.25477) 367 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:17.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:16 smithi007 bash[65864]: cluster 2023-12-16T07:52:15.872455+0000 mgr.y (mgr.25477) 368 : cluster 0 pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:17.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:16 smithi007 bash[64479]: cluster 2023-12-16T07:52:15.872455+0000 mgr.y (mgr.25477) 368 : cluster 0 pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:17.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:16 smithi169 bash[51638]: cluster 2023-12-16T07:52:15.872455+0000 mgr.y (mgr.25477) 368 : cluster 0 pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:19.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:18 smithi007 bash[64479]: cluster 2023-12-16T07:52:17.873630+0000 mgr.y (mgr.25477) 369 : cluster 0 pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:19.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:18 smithi007 bash[65864]: cluster 2023-12-16T07:52:17.873630+0000 mgr.y (mgr.25477) 369 : cluster 0 pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:19.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:18 smithi169 bash[51638]: cluster 2023-12-16T07:52:17.873630+0000 mgr.y (mgr.25477) 369 : cluster 0 pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:20.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:52:20 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:52:20] "GET /metrics HTTP/1.1" 200 38511 "" "Prometheus/2.33.4" 2023-12-16T07:52:21.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:20 smithi007 bash[64479]: cluster 2023-12-16T07:52:19.875112+0000 mgr.y (mgr.25477) 370 : cluster 0 pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:21.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:20 smithi007 bash[65864]: cluster 2023-12-16T07:52:19.875112+0000 mgr.y (mgr.25477) 370 : cluster 0 pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:21.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:20 smithi169 bash[51638]: cluster 2023-12-16T07:52:19.875112+0000 mgr.y (mgr.25477) 370 : cluster 0 pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:23.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:22 smithi007 bash[65864]: cluster 2023-12-16T07:52:21.876229+0000 mgr.y (mgr.25477) 371 : cluster 0 pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:23.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:22 smithi007 bash[64479]: cluster 2023-12-16T07:52:21.876229+0000 mgr.y (mgr.25477) 371 : cluster 0 pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:22 smithi169 bash[51638]: cluster 2023-12-16T07:52:21.876229+0000 mgr.y (mgr.25477) 371 : cluster 0 pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:24.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:23 smithi007 bash[64479]: audit 2023-12-16T07:52:23.209071+0000 mon.a (mon.0) 622 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:52:24.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:23 smithi007 bash[65864]: audit 2023-12-16T07:52:23.209071+0000 mon.a (mon.0) 622 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:52:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:23 smithi169 bash[51638]: audit 2023-12-16T07:52:23.209071+0000 mon.a (mon.0) 622 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:52:25.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:24 smithi007 bash[64479]: cluster 2023-12-16T07:52:23.877487+0000 mgr.y (mgr.25477) 372 : cluster 0 pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:25.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:24 smithi007 bash[64479]: audit 2023-12-16T07:52:24.415015+0000 mgr.y (mgr.25477) 373 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:25.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:24 smithi007 bash[65864]: cluster 2023-12-16T07:52:23.877487+0000 mgr.y (mgr.25477) 372 : cluster 0 pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:25.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:24 smithi007 bash[65864]: audit 2023-12-16T07:52:24.415015+0000 mgr.y (mgr.25477) 373 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:24 smithi169 bash[51638]: cluster 2023-12-16T07:52:23.877487+0000 mgr.y (mgr.25477) 372 : cluster 0 pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:24 smithi169 bash[51638]: audit 2023-12-16T07:52:24.415015+0000 mgr.y (mgr.25477) 373 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:27.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:26 smithi007 bash[64479]: cluster 2023-12-16T07:52:25.878631+0000 mgr.y (mgr.25477) 374 : cluster 0 pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:27.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:26 smithi007 bash[65864]: cluster 2023-12-16T07:52:25.878631+0000 mgr.y (mgr.25477) 374 : cluster 0 pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:26 smithi169 bash[51638]: cluster 2023-12-16T07:52:25.878631+0000 mgr.y (mgr.25477) 374 : cluster 0 pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:29.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:28 smithi007 bash[64479]: cluster 2023-12-16T07:52:27.879725+0000 mgr.y (mgr.25477) 375 : cluster 0 pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:29.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:29 smithi007 bash[65864]: cluster 2023-12-16T07:52:27.879725+0000 mgr.y (mgr.25477) 375 : cluster 0 pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:29.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:29 smithi169 bash[51638]: cluster 2023-12-16T07:52:27.879725+0000 mgr.y (mgr.25477) 375 : cluster 0 pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:30.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:52:30 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:52:30] "GET /metrics HTTP/1.1" 200 38515 "" "Prometheus/2.33.4" 2023-12-16T07:52:31.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:31 smithi007 bash[64479]: cluster 2023-12-16T07:52:29.881143+0000 mgr.y (mgr.25477) 376 : cluster 0 pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:31.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:31 smithi007 bash[65864]: cluster 2023-12-16T07:52:29.881143+0000 mgr.y (mgr.25477) 376 : cluster 0 pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:31.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:31 smithi169 bash[51638]: cluster 2023-12-16T07:52:29.881143+0000 mgr.y (mgr.25477) 376 : cluster 0 pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:33.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:33 smithi007 bash[64479]: cluster 2023-12-16T07:52:31.882245+0000 mgr.y (mgr.25477) 377 : cluster 0 pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:33.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:33 smithi007 bash[65864]: cluster 2023-12-16T07:52:31.882245+0000 mgr.y (mgr.25477) 377 : cluster 0 pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:33.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:33 smithi169 bash[51638]: cluster 2023-12-16T07:52:31.882245+0000 mgr.y (mgr.25477) 377 : cluster 0 pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:35.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:35 smithi007 bash[65864]: cluster 2023-12-16T07:52:33.883472+0000 mgr.y (mgr.25477) 378 : cluster 0 pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:35.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:35 smithi007 bash[65864]: audit 2023-12-16T07:52:34.425842+0000 mgr.y (mgr.25477) 379 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:35.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:35 smithi007 bash[64479]: cluster 2023-12-16T07:52:33.883472+0000 mgr.y (mgr.25477) 378 : cluster 0 pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:35.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:35 smithi007 bash[64479]: audit 2023-12-16T07:52:34.425842+0000 mgr.y (mgr.25477) 379 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:35.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:35 smithi169 bash[51638]: cluster 2023-12-16T07:52:33.883472+0000 mgr.y (mgr.25477) 378 : cluster 0 pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:35.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:35 smithi169 bash[51638]: audit 2023-12-16T07:52:34.425842+0000 mgr.y (mgr.25477) 379 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:37.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:37 smithi007 bash[65864]: cluster 2023-12-16T07:52:35.884605+0000 mgr.y (mgr.25477) 380 : cluster 0 pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:37.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:37 smithi007 bash[64479]: cluster 2023-12-16T07:52:35.884605+0000 mgr.y (mgr.25477) 380 : cluster 0 pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:37.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:37 smithi169 bash[51638]: cluster 2023-12-16T07:52:35.884605+0000 mgr.y (mgr.25477) 380 : cluster 0 pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:39.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:39 smithi007 bash[64479]: cluster 2023-12-16T07:52:37.885730+0000 mgr.y (mgr.25477) 381 : cluster 0 pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:39.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:39 smithi007 bash[64479]: audit 2023-12-16T07:52:38.209570+0000 mon.a (mon.0) 623 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:52:39.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:39 smithi007 bash[65864]: cluster 2023-12-16T07:52:37.885730+0000 mgr.y (mgr.25477) 381 : cluster 0 pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:39.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:39 smithi007 bash[65864]: audit 2023-12-16T07:52:38.209570+0000 mon.a (mon.0) 623 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:52:39.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:39 smithi169 bash[51638]: cluster 2023-12-16T07:52:37.885730+0000 mgr.y (mgr.25477) 381 : cluster 0 pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:39.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:39 smithi169 bash[51638]: audit 2023-12-16T07:52:38.209570+0000 mon.a (mon.0) 623 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:52:40.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:52:40 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:52:40] "GET /metrics HTTP/1.1" 200 38511 "" "Prometheus/2.33.4" 2023-12-16T07:52:41.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:41 smithi007 bash[64479]: cluster 2023-12-16T07:52:39.887195+0000 mgr.y (mgr.25477) 382 : cluster 0 pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:41.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:41 smithi007 bash[65864]: cluster 2023-12-16T07:52:39.887195+0000 mgr.y (mgr.25477) 382 : cluster 0 pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:41.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:41 smithi169 bash[51638]: cluster 2023-12-16T07:52:39.887195+0000 mgr.y (mgr.25477) 382 : cluster 0 pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:43.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:43 smithi007 bash[64479]: cluster 2023-12-16T07:52:41.888467+0000 mgr.y (mgr.25477) 383 : cluster 0 pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:43.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:43 smithi007 bash[65864]: cluster 2023-12-16T07:52:41.888467+0000 mgr.y (mgr.25477) 383 : cluster 0 pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:43.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:43 smithi169 bash[51638]: cluster 2023-12-16T07:52:41.888467+0000 mgr.y (mgr.25477) 383 : cluster 0 pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:45.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:45 smithi007 bash[64479]: cluster 2023-12-16T07:52:43.889870+0000 mgr.y (mgr.25477) 384 : cluster 0 pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:45.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:45 smithi007 bash[64479]: audit 2023-12-16T07:52:44.435248+0000 mgr.y (mgr.25477) 385 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:45.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:45 smithi007 bash[65864]: cluster 2023-12-16T07:52:43.889870+0000 mgr.y (mgr.25477) 384 : cluster 0 pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:45.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:45 smithi007 bash[65864]: audit 2023-12-16T07:52:44.435248+0000 mgr.y (mgr.25477) 385 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:45.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:45 smithi169 bash[51638]: cluster 2023-12-16T07:52:43.889870+0000 mgr.y (mgr.25477) 384 : cluster 0 pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:45.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:45 smithi169 bash[51638]: audit 2023-12-16T07:52:44.435248+0000 mgr.y (mgr.25477) 385 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:47.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:47 smithi007 bash[64479]: cluster 2023-12-16T07:52:45.891096+0000 mgr.y (mgr.25477) 386 : cluster 0 pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:47.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:47 smithi007 bash[65864]: cluster 2023-12-16T07:52:45.891096+0000 mgr.y (mgr.25477) 386 : cluster 0 pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:47.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:47 smithi169 bash[51638]: cluster 2023-12-16T07:52:45.891096+0000 mgr.y (mgr.25477) 386 : cluster 0 pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:49.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:49 smithi007 bash[64479]: cluster 2023-12-16T07:52:47.892268+0000 mgr.y (mgr.25477) 387 : cluster 0 pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:49.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:49 smithi007 bash[65864]: cluster 2023-12-16T07:52:47.892268+0000 mgr.y (mgr.25477) 387 : cluster 0 pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:49.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:49 smithi169 bash[51638]: cluster 2023-12-16T07:52:47.892268+0000 mgr.y (mgr.25477) 387 : cluster 0 pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:50.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:52:50 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:52:50] "GET /metrics HTTP/1.1" 200 38511 "" "Prometheus/2.33.4" 2023-12-16T07:52:51.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:51 smithi007 bash[64479]: cluster 2023-12-16T07:52:49.893679+0000 mgr.y (mgr.25477) 388 : cluster 0 pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:51.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:51 smithi007 bash[65864]: cluster 2023-12-16T07:52:49.893679+0000 mgr.y (mgr.25477) 388 : cluster 0 pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:51.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:51 smithi169 bash[51638]: cluster 2023-12-16T07:52:49.893679+0000 mgr.y (mgr.25477) 388 : cluster 0 pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:53 smithi007 bash[64479]: cluster 2023-12-16T07:52:51.894787+0000 mgr.y (mgr.25477) 389 : cluster 0 pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:53 smithi007 bash[64479]: audit 2023-12-16T07:52:52.641746+0000 mon.a (mon.0) 624 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:52:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:53 smithi007 bash[64479]: audit 2023-12-16T07:52:52.643475+0000 mon.a (mon.0) 625 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:52:53.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:53 smithi007 bash[64479]: audit 2023-12-16T07:52:52.653430+0000 mon.a (mon.0) 626 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:52:53.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:53 smithi007 bash[65864]: cluster 2023-12-16T07:52:51.894787+0000 mgr.y (mgr.25477) 389 : cluster 0 pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:53.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:53 smithi007 bash[65864]: audit 2023-12-16T07:52:52.641746+0000 mon.a (mon.0) 624 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:52:53.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:53 smithi007 bash[65864]: audit 2023-12-16T07:52:52.643475+0000 mon.a (mon.0) 625 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:52:53.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:53 smithi007 bash[65864]: audit 2023-12-16T07:52:52.653430+0000 mon.a (mon.0) 626 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:52:53.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:53 smithi169 bash[51638]: cluster 2023-12-16T07:52:51.894787+0000 mgr.y (mgr.25477) 389 : cluster 0 pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:53.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:53 smithi169 bash[51638]: audit 2023-12-16T07:52:52.641746+0000 mon.a (mon.0) 624 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-16T07:52:53.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:53 smithi169 bash[51638]: audit 2023-12-16T07:52:52.643475+0000 mon.a (mon.0) 625 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-12-16T07:52:53.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:53 smithi169 bash[51638]: audit 2023-12-16T07:52:52.653430+0000 mon.a (mon.0) 626 : audit 1 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' 2023-12-16T07:52:54.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:54 smithi007 bash[64479]: audit 2023-12-16T07:52:53.210028+0000 mon.a (mon.0) 627 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:52:54.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:54 smithi007 bash[65864]: audit 2023-12-16T07:52:53.210028+0000 mon.a (mon.0) 627 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:52:54.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:54 smithi169 bash[51638]: audit 2023-12-16T07:52:53.210028+0000 mon.a (mon.0) 627 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:52:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:55 smithi007 bash[64479]: cluster 2023-12-16T07:52:53.896113+0000 mgr.y (mgr.25477) 390 : cluster 0 pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:55.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:55 smithi007 bash[64479]: audit 2023-12-16T07:52:54.446493+0000 mgr.y (mgr.25477) 391 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:55.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:55 smithi007 bash[65864]: cluster 2023-12-16T07:52:53.896113+0000 mgr.y (mgr.25477) 390 : cluster 0 pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:55.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:55 smithi007 bash[65864]: audit 2023-12-16T07:52:54.446493+0000 mgr.y (mgr.25477) 391 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:55.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:55 smithi169 bash[51638]: cluster 2023-12-16T07:52:53.896113+0000 mgr.y (mgr.25477) 390 : cluster 0 pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:55.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:55 smithi169 bash[51638]: audit 2023-12-16T07:52:54.446493+0000 mgr.y (mgr.25477) 391 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:52:57.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:57 smithi169 bash[51638]: cluster 2023-12-16T07:52:55.897285+0000 mgr.y (mgr.25477) 392 : cluster 0 pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:57.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:57 smithi007 bash[64479]: cluster 2023-12-16T07:52:55.897285+0000 mgr.y (mgr.25477) 392 : cluster 0 pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:57.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:57 smithi007 bash[65864]: cluster 2023-12-16T07:52:55.897285+0000 mgr.y (mgr.25477) 392 : cluster 0 pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:52:59.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:52:59 smithi169 bash[51638]: cluster 2023-12-16T07:52:57.898423+0000 mgr.y (mgr.25477) 393 : cluster 0 pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:59.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:52:59 smithi007 bash[64479]: cluster 2023-12-16T07:52:57.898423+0000 mgr.y (mgr.25477) 393 : cluster 0 pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:52:59.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:52:59 smithi007 bash[65864]: cluster 2023-12-16T07:52:57.898423+0000 mgr.y (mgr.25477) 393 : cluster 0 pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:00.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:53:00 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:53:00] "GET /metrics HTTP/1.1" 200 38512 "" "Prometheus/2.33.4" 2023-12-16T07:53:01.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:01 smithi169 bash[51638]: cluster 2023-12-16T07:52:59.899956+0000 mgr.y (mgr.25477) 394 : cluster 0 pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:01.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:01 smithi007 bash[65864]: cluster 2023-12-16T07:52:59.899956+0000 mgr.y (mgr.25477) 394 : cluster 0 pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:01.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:01 smithi007 bash[64479]: cluster 2023-12-16T07:52:59.899956+0000 mgr.y (mgr.25477) 394 : cluster 0 pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:03.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:03 smithi169 bash[51638]: cluster 2023-12-16T07:53:01.901109+0000 mgr.y (mgr.25477) 395 : cluster 0 pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:03.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:03 smithi007 bash[64479]: cluster 2023-12-16T07:53:01.901109+0000 mgr.y (mgr.25477) 395 : cluster 0 pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:03.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:03 smithi007 bash[65864]: cluster 2023-12-16T07:53:01.901109+0000 mgr.y (mgr.25477) 395 : cluster 0 pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:05.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:05 smithi169 bash[51638]: cluster 2023-12-16T07:53:03.902370+0000 mgr.y (mgr.25477) 396 : cluster 0 pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:05.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:05 smithi169 bash[51638]: audit 2023-12-16T07:53:04.457700+0000 mgr.y (mgr.25477) 397 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:53:05.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:05 smithi007 bash[64479]: cluster 2023-12-16T07:53:03.902370+0000 mgr.y (mgr.25477) 396 : cluster 0 pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:05.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:05 smithi007 bash[64479]: audit 2023-12-16T07:53:04.457700+0000 mgr.y (mgr.25477) 397 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:53:05.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:05 smithi007 bash[65864]: cluster 2023-12-16T07:53:03.902370+0000 mgr.y (mgr.25477) 396 : cluster 0 pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:05.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:05 smithi007 bash[65864]: audit 2023-12-16T07:53:04.457700+0000 mgr.y (mgr.25477) 397 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:53:07.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:07 smithi169 bash[51638]: cluster 2023-12-16T07:53:05.903519+0000 mgr.y (mgr.25477) 398 : cluster 0 pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:07.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:07 smithi007 bash[64479]: cluster 2023-12-16T07:53:05.903519+0000 mgr.y (mgr.25477) 398 : cluster 0 pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:07.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:07 smithi007 bash[65864]: cluster 2023-12-16T07:53:05.903519+0000 mgr.y (mgr.25477) 398 : cluster 0 pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:09.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:09 smithi169 bash[51638]: cluster 2023-12-16T07:53:07.904709+0000 mgr.y (mgr.25477) 399 : cluster 0 pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:09.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:09 smithi169 bash[51638]: audit 2023-12-16T07:53:08.210421+0000 mon.a (mon.0) 628 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:53:09.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:09 smithi007 bash[64479]: cluster 2023-12-16T07:53:07.904709+0000 mgr.y (mgr.25477) 399 : cluster 0 pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:09.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:09 smithi007 bash[64479]: audit 2023-12-16T07:53:08.210421+0000 mon.a (mon.0) 628 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:53:09.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:09 smithi007 bash[65864]: cluster 2023-12-16T07:53:07.904709+0000 mgr.y (mgr.25477) 399 : cluster 0 pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:09.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:09 smithi007 bash[65864]: audit 2023-12-16T07:53:08.210421+0000 mon.a (mon.0) 628 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:53:10.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:53:10 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:53:10] "GET /metrics HTTP/1.1" 200 38509 "" "Prometheus/2.33.4" 2023-12-16T07:53:11.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:11 smithi169 bash[51638]: cluster 2023-12-16T07:53:09.906137+0000 mgr.y (mgr.25477) 400 : cluster 0 pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:11.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:11 smithi007 bash[64479]: cluster 2023-12-16T07:53:09.906137+0000 mgr.y (mgr.25477) 400 : cluster 0 pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:11.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:11 smithi007 bash[65864]: cluster 2023-12-16T07:53:09.906137+0000 mgr.y (mgr.25477) 400 : cluster 0 pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:12.723 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph orch ps' 2023-12-16T07:53:13.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:13 smithi007 bash[65864]: cluster 2023-12-16T07:53:11.907189+0000 mgr.y (mgr.25477) 401 : cluster 0 pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:13.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:13 smithi007 bash[64479]: cluster 2023-12-16T07:53:11.907189+0000 mgr.y (mgr.25477) 401 : cluster 0 pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:13.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:13 smithi169 bash[51638]: cluster 2023-12-16T07:53:11.907189+0000 mgr.y (mgr.25477) 401 : cluster 0 pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:15.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:15 smithi169 bash[51638]: cluster 2023-12-16T07:53:13.908659+0000 mgr.y (mgr.25477) 402 : cluster 0 pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:15.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:15 smithi169 bash[51638]: audit 2023-12-16T07:53:14.467205+0000 mgr.y (mgr.25477) 403 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:53:15.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:15 smithi007 bash[64479]: cluster 2023-12-16T07:53:13.908659+0000 mgr.y (mgr.25477) 402 : cluster 0 pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:15.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:15 smithi007 bash[64479]: audit 2023-12-16T07:53:14.467205+0000 mgr.y (mgr.25477) 403 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:53:15.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:15 smithi007 bash[65864]: cluster 2023-12-16T07:53:13.908659+0000 mgr.y (mgr.25477) 402 : cluster 0 pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:15.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:15 smithi007 bash[65864]: audit 2023-12-16T07:53:14.467205+0000 mgr.y (mgr.25477) 403 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:53:16.087 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-12-16T07:53:16.087 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (9m) 3m ago 16m 16.8M - 0.23.0 ba2b418f427c 6da6262b4f30 2023-12-16T07:53:16.087 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi169 *:3000 running (8m) 95s ago 16m 44.7M - 8.3.5 dad864ee21e9 1481acb31e9c 2023-12-16T07:53:16.087 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.crrdaq smithi007 running (8m) 3m ago 14m 47.7M - 3.5 e1d6a67b021e 8d8009d0c83f 2023-12-16T07:53:16.087 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi169 *:8443,9283,8765 running (7m) 95s ago 22m 436M - 19.0.0-191-g6d933094 a29c258b5d6f 5295ba35c2cd 2023-12-16T07:53:16.087 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (8m) 3m ago 24m 493M - 19.0.0-191-g6d933094 a29c258b5d6f 765072a9ccc7 2023-12-16T07:53:16.087 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (7m) 3m ago 25m 52.7M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 5a266b45f3ce 2023-12-16T07:53:16.087 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi169 running (6m) 95s ago 23m 45.5M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1fc9bf864dda 2023-12-16T07:53:16.087 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (6m) 3m ago 23m 42.0M 2048M 19.0.0-191-g6d933094 a29c258b5d6f 1dc91c31fc27 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (9m) 3m ago 17m 13.0M - 1.3.1 1dbe0e931976 c04593605e31 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi169 *:9100 running (9m) 95s ago 17m 15.2M - 1.3.1 1dbe0e931976 361882333d2b 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (5m) 3m ago 22m 52.0M 2782M 19.0.0-191-g6d933094 a29c258b5d6f a00724c6d077 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (4m) 3m ago 21m 51.3M 2782M 19.0.0-191-g6d933094 a29c258b5d6f 5e90fce88079 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (4m) 3m ago 21m 46.9M 2782M 19.0.0-191-g6d933094 a29c258b5d6f 906a30291e88 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (3m) 3m ago 20m - 2782M 19.0.0-191-g6d933094 a29c258b5d6f 8e5e7d6ede0f 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi169 running (3m) 95s ago 19m 52.9M 3037M 19.0.0-191-g6d933094 a29c258b5d6f b612e0176579 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi169 running (2m) 95s ago 19m 49.6M 3037M 19.0.0-191-g6d933094 a29c258b5d6f c1ca50cfe550 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi169 running (2m) 95s ago 18m 48.2M 3037M 19.0.0-191-g6d933094 a29c258b5d6f b335dc393bbe 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi169 running (99s) 95s ago 18m - 3037M 19.0.0-191-g6d933094 a29c258b5d6f 0914497c21be 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi169 *:9095 running (8m) 95s ago 17m 64.6M - 2.33.4 514e6a882f6e 2652ce1b8540 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.mglnhl smithi007 *:8000 running (14m) 3m ago 14m 88.1M - 17.2.0 e1d6a67b021e 73ec19de719f 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi169.okfkhg smithi169 *:8000 running (14m) 95s ago 14m 88.9M - 17.2.0 e1d6a67b021e 28bf28384f61 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.tizuzt smithi007 *:80 running (14m) 3m ago 14m 88.0M - 17.2.0 e1d6a67b021e 766edd790f04 2023-12-16T07:53:16.088 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi169.aiyfmr smithi169 *:80 running (14m) 95s ago 14m 88.7M - 17.2.0 e1d6a67b021e dd583b735ee0 2023-12-16T07:53:16.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph versions' 2023-12-16T07:53:17.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:17 smithi007 bash[64479]: cluster 2023-12-16T07:53:15.909447+0000 mgr.y (mgr.25477) 404 : cluster 0 pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:17.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:17 smithi007 bash[64479]: audit 2023-12-16T07:53:16.060562+0000 mgr.y (mgr.25477) 405 : audit 0 from='client.34593 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:53:17.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:17 smithi007 bash[65864]: cluster 2023-12-16T07:53:15.909447+0000 mgr.y (mgr.25477) 404 : cluster 0 pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:17.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:17 smithi007 bash[65864]: audit 2023-12-16T07:53:16.060562+0000 mgr.y (mgr.25477) 405 : audit 0 from='client.34593 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:53:17.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:17 smithi169 bash[51638]: cluster 2023-12-16T07:53:15.909447+0000 mgr.y (mgr.25477) 404 : cluster 0 pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:17.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:17 smithi169 bash[51638]: audit 2023-12-16T07:53:16.060562+0000 mgr.y (mgr.25477) 405 : audit 0 from='client.34593 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:53:19.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:19 smithi169 bash[51638]: cluster 2023-12-16T07:53:17.910732+0000 mgr.y (mgr.25477) 406 : cluster 0 pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:19.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:19 smithi007 bash[65864]: cluster 2023-12-16T07:53:17.910732+0000 mgr.y (mgr.25477) 406 : cluster 0 pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:19.498 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:19 smithi007 bash[64479]: cluster 2023-12-16T07:53:17.910732+0000 mgr.y (mgr.25477) 406 : cluster 0 pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:20.028 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:53:20.029 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2023-12-16T07:53:20.029 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 3 2023-12-16T07:53:20.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:53:20.029 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2023-12-16T07:53:20.029 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 2 2023-12-16T07:53:20.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:53:20.029 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2023-12-16T07:53:20.029 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 8 2023-12-16T07:53:20.029 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:53:20.029 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2023-12-16T07:53:20.030 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2023-12-16T07:53:20.030 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2023-12-16T07:53:20.030 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2023-12-16T07:53:20.030 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2023-12-16T07:53:20.030 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-191-g6d933094 (6d933094a156ec7587742e5aa25fba5ba27ec678) reef (dev)": 13 2023-12-16T07:53:20.030 INFO:teuthology.orchestra.run.smithi007.stdout: } 2023-12-16T07:53:20.030 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:53:20.318 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:53:20 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:53:20] "GET /metrics HTTP/1.1" 200 38509 "" "Prometheus/2.33.4" 2023-12-16T07:53:20.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:20 smithi007 bash[65864]: audit 2023-12-16T07:53:20.024922+0000 mon.c (mon.1) 15 : audit 0 from='client.? 172.21.15.7:0/1530889169' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:53:20.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:20 smithi007 bash[64479]: audit 2023-12-16T07:53:20.024922+0000 mon.c (mon.1) 15 : audit 0 from='client.? 172.21.15.7:0/1530889169' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:53:20.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:20 smithi169 bash[51638]: audit 2023-12-16T07:53:20.024922+0000 mon.c (mon.1) 15 : audit 0 from='client.? 172.21.15.7:0/1530889169' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:53:20.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph orch upgrade status' 2023-12-16T07:53:21.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:21 smithi007 bash[64479]: cluster 2023-12-16T07:53:19.911696+0000 mgr.y (mgr.25477) 407 : cluster 0 pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:21.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:21 smithi007 bash[65864]: cluster 2023-12-16T07:53:19.911696+0000 mgr.y (mgr.25477) 407 : cluster 0 pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:21.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:21 smithi169 bash[51638]: cluster 2023-12-16T07:53:19.911696+0000 mgr.y (mgr.25477) 407 : cluster 0 pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:23.406 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:23 smithi007 bash[65864]: cluster 2023-12-16T07:53:21.912776+0000 mgr.y (mgr.25477) 408 : cluster 0 pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:23.406 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:23 smithi007 bash[64479]: cluster 2023-12-16T07:53:21.912776+0000 mgr.y (mgr.25477) 408 : cluster 0 pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:23.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:23 smithi169 bash[51638]: cluster 2023-12-16T07:53:21.912776+0000 mgr.y (mgr.25477) 408 : cluster 0 pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:23.859 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2023-12-16T07:53:23.859 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:6d933094a156ec7587742e5aa25fba5ba27ec678", 2023-12-16T07:53:23.859 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2023-12-16T07:53:23.859 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2023-12-16T07:53:23.859 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2023-12-16T07:53:23.859 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr", 2023-12-16T07:53:23.859 INFO:teuthology.orchestra.run.smithi007.stdout: "osd", 2023-12-16T07:53:23.859 INFO:teuthology.orchestra.run.smithi007.stdout: "mon" 2023-12-16T07:53:23.859 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2023-12-16T07:53:23.859 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "13/23 daemons upgraded", 2023-12-16T07:53:23.859 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", 2023-12-16T07:53:23.859 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": true 2023-12-16T07:53:23.860 INFO:teuthology.orchestra.run.smithi007.stdout:} 2023-12-16T07:53:24.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:24 smithi007 bash[64479]: audit 2023-12-16T07:53:23.210941+0000 mon.a (mon.0) 629 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:53:24.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:24 smithi007 bash[65864]: audit 2023-12-16T07:53:23.210941+0000 mon.a (mon.0) 629 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:53:24.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:24 smithi169 bash[51638]: audit 2023-12-16T07:53:23.210941+0000 mon.a (mon.0) 629 : audit 0 from='mgr.25477 172.21.15.7:0/1608621280' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2023-12-16T07:53:24.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph health detail' 2023-12-16T07:53:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:25 smithi169 bash[51638]: audit 2023-12-16T07:53:23.855212+0000 mgr.y (mgr.25477) 409 : audit 0 from='client.44536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:53:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:25 smithi169 bash[51638]: cluster 2023-12-16T07:53:23.913792+0000 mgr.y (mgr.25477) 410 : cluster 0 pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:25.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:25 smithi169 bash[51638]: audit 2023-12-16T07:53:24.475162+0000 mgr.y (mgr.25477) 411 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:53:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:25 smithi007 bash[64479]: audit 2023-12-16T07:53:23.855212+0000 mgr.y (mgr.25477) 409 : audit 0 from='client.44536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:53:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:25 smithi007 bash[64479]: cluster 2023-12-16T07:53:23.913792+0000 mgr.y (mgr.25477) 410 : cluster 0 pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:25.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:25 smithi007 bash[64479]: audit 2023-12-16T07:53:24.475162+0000 mgr.y (mgr.25477) 411 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:53:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:25 smithi007 bash[65864]: audit 2023-12-16T07:53:23.855212+0000 mgr.y (mgr.25477) 409 : audit 0 from='client.44536 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2023-12-16T07:53:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:25 smithi007 bash[65864]: cluster 2023-12-16T07:53:23.913792+0000 mgr.y (mgr.25477) 410 : cluster 0 pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:25.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:25 smithi007 bash[65864]: audit 2023-12-16T07:53:24.475162+0000 mgr.y (mgr.25477) 411 : audit 0 from='client.15513 -' entity='client.iscsi.foo.smithi007.crrdaq' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2023-12-16T07:53:27.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:27 smithi169 bash[51638]: cluster 2023-12-16T07:53:25.914866+0000 mgr.y (mgr.25477) 412 : cluster 0 pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:27.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:27 smithi007 bash[65864]: cluster 2023-12-16T07:53:25.914866+0000 mgr.y (mgr.25477) 412 : cluster 0 pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:27.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:27 smithi007 bash[64479]: cluster 2023-12-16T07:53:25.914866+0000 mgr.y (mgr.25477) 412 : cluster 0 pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:27.757 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_ERR Upgrade: failed due to an unexpected exception 2023-12-16T07:53:27.757 INFO:teuthology.orchestra.run.smithi007.stdout:[ERR] UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception 2023-12-16T07:53:27.758 INFO:teuthology.orchestra.run.smithi007.stdout: Unexpected exception occurred during upgrade process: osd require-osd-release failed: unrecognized release unknown retval: -22 2023-12-16T07:53:28.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:28 smithi007 bash[65864]: audit 2023-12-16T07:53:27.754253+0000 mon.a (mon.0) 630 : audit 0 from='client.? 172.21.15.7:0/1949656827' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:53:28.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:28 smithi007 bash[64479]: audit 2023-12-16T07:53:27.754253+0000 mon.a (mon.0) 630 : audit 0 from='client.? 172.21.15.7:0/1949656827' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:53:28.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:28 smithi169 bash[51638]: audit 2023-12-16T07:53:27.754253+0000 mon.a (mon.0) 630 : audit 0 from='client.? 172.21.15.7:0/1949656827' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2023-12-16T07:53:28.466 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2023-12-16T07:53:29.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:29 smithi169 bash[51638]: cluster 2023-12-16T07:53:27.916052+0000 mgr.y (mgr.25477) 413 : cluster 0 pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:29.502 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:29 smithi007 bash[64479]: cluster 2023-12-16T07:53:27.916052+0000 mgr.y (mgr.25477) 413 : cluster 0 pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:29.502 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:29 smithi007 bash[65864]: cluster 2023-12-16T07:53:27.916052+0000 mgr.y (mgr.25477) 413 : cluster 0 pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2023-12-16T07:53:30.568 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:53:30 smithi007 bash[61260]: ::ffff:172.21.15.169 - - [16/Dec/2023:07:53:30] "GET /metrics HTTP/1.1" 200 38509 "" "Prometheus/2.33.4" 2023-12-16T07:53:31.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:31 smithi007 bash[64479]: cluster 2023-12-16T07:53:29.917558+0000 mgr.y (mgr.25477) 414 : cluster 0 pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:31.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:31 smithi007 bash[65864]: cluster 2023-12-16T07:53:29.917558+0000 mgr.y (mgr.25477) 414 : cluster 0 pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:31.433 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:31 smithi169 bash[51638]: cluster 2023-12-16T07:53:29.917558+0000 mgr.y (mgr.25477) 414 : cluster 0 pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2023-12-16T07:53:31.601 INFO:teuthology.orchestra.run.smithi007.stdout:false 2023-12-16T07:53:32.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T07:53:32.333 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/cephadm.py", line 1351, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-12-16T07:53:32.409 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8136ccd4f2ae4b8aa737eb1184bdf6da Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/cephadm.py", line 1351, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_ceph_ceph-c_6d933094a156ec7587742e5aa25fba5ba27ec678/qa/tasks/cephadm.py", line 40, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2023-12-16T07:53:32.411 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-12-16T07:53:32.634 INFO:tasks.cephadm:Teardown begin 2023-12-16T07:53:32.635 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-12-16T07:53:32.649 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:32 smithi169 bash[51638]: audit 2023-12-16T07:53:31.580719+0000 mon.a (mon.0) 631 : audit 0 from='client.? 172.21.15.7:0/160217292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:53:32.650 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:32 smithi007 bash[64479]: audit 2023-12-16T07:53:31.580719+0000 mon.a (mon.0) 631 : audit 0 from='client.? 172.21.15.7:0/160217292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:53:32.650 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:32 smithi007 bash[65864]: audit 2023-12-16T07:53:31.580719+0000 mon.a (mon.0) 631 : audit 0 from='client.? 172.21.15.7:0/160217292' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-12-16T07:53:32.667 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-12-16T07:53:32.686 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-12-16T07:53:32.686 DEBUG:teuthology.orchestra.run.smithi007:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-16T07:53:32.715 DEBUG:teuthology.orchestra.run.smithi169:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-16T07:53:32.736 INFO:tasks.cephadm:Stopping all daemons... 2023-12-16T07:53:32.736 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-12-16T07:53:32.736 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.a 2023-12-16T07:53:32.924 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:32 smithi007 systemd[1]: Stopping Ceph mon.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:53:32.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:32 smithi007 bash[64479]: debug 2023-12-16T07:53:32.895+0000 7f4f5f711700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-16T07:53:32.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Dec 16 07:53:32 smithi007 bash[64479]: debug 2023-12-16T07:53:32.895+0000 7f4f5f711700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2023-12-16T07:53:33.179 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:53:32 smithi007 bash[61260]: [16/Dec/2023:07:53:32] ENGINE Bus STOPPING 2023-12-16T07:53:33.179 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:53:32 smithi007 bash[61260]: [16/Dec/2023:07:53:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2023-12-16T07:53:33.179 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:53:32 smithi007 bash[61260]: [16/Dec/2023:07:53:32] ENGINE Bus STOPPED 2023-12-16T07:53:33.179 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:53:32 smithi007 bash[61260]: [16/Dec/2023:07:53:32] ENGINE Bus STARTING 2023-12-16T07:53:33.179 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:53:33 smithi007 bash[61260]: [16/Dec/2023:07:53:33] ENGINE Serving on http://:::9283 2023-12-16T07:53:33.180 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:53:33 smithi007 bash[61260]: [16/Dec/2023:07:53:33] ENGINE Bus STARTED 2023-12-16T07:53:33.779 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.a.service' 2023-12-16T07:53:33.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T07:53:33.813 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-12-16T07:53:33.814 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2023-12-16T07:53:33.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.c 2023-12-16T07:53:34.175 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:33 smithi007 systemd[1]: Stopping Ceph mon.c for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:53:34.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:34 smithi007 bash[65864]: debug 2023-12-16T07:53:34.015+0000 7f7241044700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-16T07:53:34.176 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:34 smithi007 bash[65864]: debug 2023-12-16T07:53:34.015+0000 7f7241044700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2023-12-16T07:53:34.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Dec 16 07:53:34 smithi007 bash[78835]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-mon-c 2023-12-16T07:53:34.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.c.service' 2023-12-16T07:53:34.943 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T07:53:34.944 INFO:tasks.cephadm.mon.b:Stopped mon.c 2023-12-16T07:53:34.944 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2023-12-16T07:53:34.945 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.b 2023-12-16T07:53:35.364 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:34 smithi169 systemd[1]: Stopping Ceph mon.b for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:53:35.364 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:35 smithi169 bash[51638]: debug 2023-12-16T07:53:35.119+0000 7f6c56a37700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-16T07:53:35.364 INFO:journalctl@ceph.mon.b.smithi169.stdout:Dec 16 07:53:35 smithi169 bash[51638]: debug 2023-12-16T07:53:35.119+0000 7f6c56a37700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2023-12-16T07:53:36.160 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mon.b.service' 2023-12-16T07:53:36.205 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T07:53:36.205 INFO:tasks.cephadm.mon.b:Stopped mon.b 2023-12-16T07:53:36.205 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-12-16T07:53:36.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.y 2023-12-16T07:53:36.561 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:53:36 smithi007 systemd[1]: Stopping Ceph mgr.y for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:53:36.561 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Dec 16 07:53:36 smithi007 bash[61260]: debug 2023-12-16T07:53:36.359+0000 7f2d1f3bd700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:53:37.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.y.service' 2023-12-16T07:53:37.174 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T07:53:37.174 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-12-16T07:53:37.174 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-12-16T07:53:37.174 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.x 2023-12-16T07:53:37.559 INFO:journalctl@ceph.mgr.x.smithi169.stdout:Dec 16 07:53:37 smithi169 systemd[1]: Stopping Ceph mgr.x for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:53:38.175 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@mgr.x.service' 2023-12-16T07:53:38.205 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T07:53:38.205 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-12-16T07:53:38.205 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-12-16T07:53:38.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.0 2023-12-16T07:53:38.568 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:53:38 smithi007 systemd[1]: Stopping Ceph osd.0 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:53:38.568 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:53:38 smithi007 bash[68738]: debug 2023-12-16T07:53:38.395+0000 7fd696da0700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:53:38.569 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:53:38 smithi007 bash[68738]: debug 2023-12-16T07:53:38.395+0000 7fd696da0700 -1 osd.0 143 *** Got signal Terminated *** 2023-12-16T07:53:38.569 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:53:38 smithi007 bash[68738]: debug 2023-12-16T07:53:38.395+0000 7fd696da0700 -1 osd.0 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:53:44.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:53:43 smithi007 bash[59950]: level=warn ts=2023-12-16T07:53:43.753Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:53:44.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:53:43 smithi007 bash[59950]: level=warn ts=2023-12-16T07:53:43.754Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi169.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused" 2023-12-16T07:53:44.814 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:53:44 smithi007 bash[79034]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-0 2023-12-16T07:53:44.814 INFO:journalctl@ceph.osd.0.smithi007.stdout:Dec 16 07:53:44 smithi007 bash[79109]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-0 2023-12-16T07:53:46.633 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.0.service' 2023-12-16T07:53:46.668 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T07:53:46.668 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-12-16T07:53:46.669 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-12-16T07:53:46.669 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.1 2023-12-16T07:53:46.829 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:53:46 smithi007 systemd[1]: Stopping Ceph osd.1 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:53:47.068 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:53:46 smithi007 bash[70898]: debug 2023-12-16T07:53:46.819+0000 7f8373a70700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:53:47.068 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:53:46 smithi007 bash[70898]: debug 2023-12-16T07:53:46.819+0000 7f8373a70700 -1 osd.1 143 *** Got signal Terminated *** 2023-12-16T07:53:47.068 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:53:46 smithi007 bash[70898]: debug 2023-12-16T07:53:46.819+0000 7f8373a70700 -1 osd.1 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:53:53.318 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:53:52 smithi007 bash[79253]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-1 2023-12-16T07:53:53.318 INFO:journalctl@ceph.osd.1.smithi007.stdout:Dec 16 07:53:53 smithi007 bash[79329]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-1 2023-12-16T07:53:54.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:53:53 smithi007 bash[59950]: level=error ts=2023-12-16T07:53:53.752Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=10 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi169.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:53:54.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:53:53 smithi007 bash[59950]: level=warn ts=2023-12-16T07:53:53.754Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:53:54.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:53:53 smithi007 bash[59950]: level=warn ts=2023-12-16T07:53:53.755Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi169.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused" 2023-12-16T07:53:54.991 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.1.service' 2023-12-16T07:53:55.027 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T07:53:55.027 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-12-16T07:53:55.027 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-12-16T07:53:55.028 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.2 2023-12-16T07:53:55.318 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:53:55 smithi007 systemd[1]: Stopping Ceph osd.2 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:53:55.318 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:53:55 smithi007 bash[73067]: debug 2023-12-16T07:53:55.182+0000 7feb2f06a700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:53:55.318 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:53:55 smithi007 bash[73067]: debug 2023-12-16T07:53:55.182+0000 7feb2f06a700 -1 osd.2 143 *** Got signal Terminated *** 2023-12-16T07:53:55.319 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:53:55 smithi007 bash[73067]: debug 2023-12-16T07:53:55.182+0000 7feb2f06a700 -1 osd.2 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:54:01.353 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:54:01 smithi007 bash[79472]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-2 2023-12-16T07:54:01.354 INFO:journalctl@ceph.osd.2.smithi007.stdout:Dec 16 07:54:01 smithi007 bash[79548]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-2 2023-12-16T07:54:03.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.2.service' 2023-12-16T07:54:03.236 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T07:54:03.237 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-12-16T07:54:03.237 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-12-16T07:54:03.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.3 2023-12-16T07:54:03.568 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:54:03 smithi007 systemd[1]: Stopping Ceph osd.3 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:54:03.568 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:54:03 smithi007 bash[75209]: debug 2023-12-16T07:54:03.374+0000 7fb6efde1700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:54:03.568 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:54:03 smithi007 bash[75209]: debug 2023-12-16T07:54:03.374+0000 7fb6efde1700 -1 osd.3 143 *** Got signal Terminated *** 2023-12-16T07:54:03.568 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:54:03 smithi007 bash[75209]: debug 2023-12-16T07:54:03.374+0000 7fb6efde1700 -1 osd.3 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:54:04.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:03 smithi007 bash[59950]: level=error ts=2023-12-16T07:54:03.753Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=10 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi169.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:54:04.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:03 smithi007 bash[59950]: level=warn ts=2023-12-16T07:54:03.754Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:54:04.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:03 smithi007 bash[59950]: level=warn ts=2023-12-16T07:54:03.756Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi169.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused" 2023-12-16T07:54:05.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:05 smithi169 bash[58112]: debug 2023-12-16T07:54:05.411+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:05.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:05 smithi169 bash[60136]: debug 2023-12-16T07:54:05.319+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:06.318 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:54:05 smithi007 bash[75209]: debug 2023-12-16T07:54:05.866+0000 7fb6e9bd7700 -1 osd.3 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:41.033342+0000 front 2023-12-16T07:53:41.033445+0000 (oldest deadline 2023-12-16T07:54:05.133566+0000) 2023-12-16T07:54:06.319 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:05 smithi169 bash[54022]: debug 2023-12-16T07:54:05.971+0000 7fc643ed3700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.017956+0000 front 2023-12-16T07:53:42.017462+0000 (oldest deadline 2023-12-16T07:54:05.517652+0000) 2023-12-16T07:54:06.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:06 smithi169 bash[60136]: debug 2023-12-16T07:54:06.319+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:06.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:06 smithi169 bash[58112]: debug 2023-12-16T07:54:06.407+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:07.183 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:06 smithi169 bash[54022]: debug 2023-12-16T07:54:06.923+0000 7fc643ed3700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.017956+0000 front 2023-12-16T07:53:42.017462+0000 (oldest deadline 2023-12-16T07:54:05.517652+0000) 2023-12-16T07:54:07.318 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:54:06 smithi007 bash[75209]: debug 2023-12-16T07:54:06.826+0000 7fb6e9bd7700 -1 osd.3 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:41.033342+0000 front 2023-12-16T07:53:41.033445+0000 (oldest deadline 2023-12-16T07:54:05.133566+0000) 2023-12-16T07:54:07.685 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:07 smithi169 bash[58112]: debug 2023-12-16T07:54:07.427+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:07.686 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:07 smithi169 bash[60136]: debug 2023-12-16T07:54:07.347+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:08.068 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:54:07 smithi007 bash[75209]: debug 2023-12-16T07:54:07.798+0000 7fb6e9bd7700 -1 osd.3 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:41.033342+0000 front 2023-12-16T07:53:41.033445+0000 (oldest deadline 2023-12-16T07:54:05.133566+0000) 2023-12-16T07:54:08.183 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:07 smithi169 bash[54022]: debug 2023-12-16T07:54:07.903+0000 7fc643ed3700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.017956+0000 front 2023-12-16T07:53:42.017462+0000 (oldest deadline 2023-12-16T07:54:05.517652+0000) 2023-12-16T07:54:08.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:08 smithi169 bash[58112]: debug 2023-12-16T07:54:08.383+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:08.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:08 smithi169 bash[60136]: debug 2023-12-16T07:54:08.339+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:09.323 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:08 smithi169 bash[54022]: debug 2023-12-16T07:54:08.943+0000 7fc643ed3700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.017956+0000 front 2023-12-16T07:53:42.017462+0000 (oldest deadline 2023-12-16T07:54:05.517652+0000) 2023-12-16T07:54:09.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:09 smithi169 bash[58112]: debug 2023-12-16T07:54:09.399+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:09.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:09 smithi169 bash[60136]: debug 2023-12-16T07:54:09.319+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:09.818 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:54:09 smithi007 bash[79693]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-3 2023-12-16T07:54:09.818 INFO:journalctl@ceph.osd.3.smithi007.stdout:Dec 16 07:54:09 smithi007 bash[79766]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-3 2023-12-16T07:54:10.183 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:09 smithi169 bash[54022]: debug 2023-12-16T07:54:09.923+0000 7fc643ed3700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.017956+0000 front 2023-12-16T07:53:42.017462+0000 (oldest deadline 2023-12-16T07:54:05.517652+0000) 2023-12-16T07:54:10.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:10 smithi169 bash[58112]: debug 2023-12-16T07:54:10.350+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:10.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:10 smithi169 bash[60136]: debug 2023-12-16T07:54:10.362+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:10.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:10 smithi169 bash[56063]: debug 2023-12-16T07:54:10.402+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:11.354 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:10 smithi169 bash[54022]: debug 2023-12-16T07:54:10.926+0000 7fc643ed3700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.017956+0000 front 2023-12-16T07:53:42.017462+0000 (oldest deadline 2023-12-16T07:54:05.517652+0000) 2023-12-16T07:54:11.359 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.3.service' 2023-12-16T07:54:11.395 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T07:54:11.395 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-12-16T07:54:11.395 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-12-16T07:54:11.395 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.4 2023-12-16T07:54:11.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:11 smithi169 bash[56063]: debug 2023-12-16T07:54:11.426+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:11.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:11 smithi169 bash[60136]: debug 2023-12-16T07:54:11.346+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:11.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:11 smithi169 bash[58112]: debug 2023-12-16T07:54:11.394+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:11.684 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:11 smithi169 systemd[1]: Stopping Ceph osd.4 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:54:11.684 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:11 smithi169 bash[54022]: debug 2023-12-16T07:54:11.570+0000 7fc6490db700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:54:11.684 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:11 smithi169 bash[54022]: debug 2023-12-16T07:54:11.570+0000 7fc6490db700 -1 osd.4 143 *** Got signal Terminated *** 2023-12-16T07:54:11.684 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:11 smithi169 bash[54022]: debug 2023-12-16T07:54:11.570+0000 7fc6490db700 -1 osd.4 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:54:12.315 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:11 smithi169 bash[54022]: debug 2023-12-16T07:54:11.966+0000 7fc643ed3700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.017956+0000 front 2023-12-16T07:53:42.017462+0000 (oldest deadline 2023-12-16T07:54:05.517652+0000) 2023-12-16T07:54:12.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:12 smithi169 bash[60136]: debug 2023-12-16T07:54:12.310+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:12.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:12 smithi169 bash[56063]: debug 2023-12-16T07:54:12.382+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:12.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:12 smithi169 bash[58112]: debug 2023-12-16T07:54:12.414+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:13.296 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:13 smithi169 bash[54022]: debug 2023-12-16T07:54:13.002+0000 7fc643ed3700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.017956+0000 front 2023-12-16T07:53:42.017462+0000 (oldest deadline 2023-12-16T07:54:05.517652+0000) 2023-12-16T07:54:13.610 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:13 smithi169 bash[58112]: debug 2023-12-16T07:54:13.370+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:13.611 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:13 smithi169 bash[60136]: debug 2023-12-16T07:54:13.290+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:13.611 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:13 smithi169 bash[56063]: debug 2023-12-16T07:54:13.346+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:13.611 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:13 smithi169 bash[56063]: debug 2023-12-16T07:54:13.346+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:49.497577+0000 front 2023-12-16T07:53:49.497519+0000 (oldest deadline 2023-12-16T07:54:12.897939+0000) 2023-12-16T07:54:13.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:13 smithi169 bash[49087]: ts=2023-12-16T07:54:13.610Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.7:8765: connect: connection refused" 2023-12-16T07:54:13.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:13 smithi169 bash[49087]: ts=2023-12-16T07:54:13.612Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.7:8765: connect: connection refused" 2023-12-16T07:54:13.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:13 smithi169 bash[49087]: ts=2023-12-16T07:54:13.615Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.7:8765: connect: connection refused" 2023-12-16T07:54:13.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:13 smithi169 bash[49087]: ts=2023-12-16T07:54:13.619Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.7:8765: connect: connection refused" 2023-12-16T07:54:14.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:13 smithi007 bash[59950]: level=error ts=2023-12-16T07:54:13.754Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=10 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi169.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:54:14.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:13 smithi007 bash[59950]: level=warn ts=2023-12-16T07:54:13.755Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:54:14.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:13 smithi007 bash[59950]: level=warn ts=2023-12-16T07:54:13.757Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi169.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused" 2023-12-16T07:54:14.269 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:13 smithi169 bash[54022]: debug 2023-12-16T07:54:13.974+0000 7fc643ed3700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.017956+0000 front 2023-12-16T07:53:42.017462+0000 (oldest deadline 2023-12-16T07:54:05.517652+0000) 2023-12-16T07:54:14.269 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:13 smithi169 bash[54022]: debug 2023-12-16T07:54:13.974+0000 7fc643ed3700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:48.920271+0000 front 2023-12-16T07:53:48.920086+0000 (oldest deadline 2023-12-16T07:54:13.020176+0000) 2023-12-16T07:54:14.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:14 smithi169 bash[60136]: debug 2023-12-16T07:54:14.262+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:14.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:14 smithi169 bash[56063]: debug 2023-12-16T07:54:14.354+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:14.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:14 smithi169 bash[56063]: debug 2023-12-16T07:54:14.354+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:49.497577+0000 front 2023-12-16T07:53:49.497519+0000 (oldest deadline 2023-12-16T07:54:12.897939+0000) 2023-12-16T07:54:14.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:14 smithi169 bash[58112]: debug 2023-12-16T07:54:14.410+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:15.310 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:14 smithi169 bash[54022]: debug 2023-12-16T07:54:14.986+0000 7fc643ed3700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.017956+0000 front 2023-12-16T07:53:42.017462+0000 (oldest deadline 2023-12-16T07:54:05.517652+0000) 2023-12-16T07:54:15.310 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:14 smithi169 bash[54022]: debug 2023-12-16T07:54:14.986+0000 7fc643ed3700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:48.920271+0000 front 2023-12-16T07:53:48.920086+0000 (oldest deadline 2023-12-16T07:54:13.020176+0000) 2023-12-16T07:54:15.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:15 smithi169 bash[58112]: debug 2023-12-16T07:54:15.418+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:15.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:15 smithi169 bash[60136]: debug 2023-12-16T07:54:15.306+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:15.684 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:15 smithi169 bash[56063]: debug 2023-12-16T07:54:15.358+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:15.684 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:15 smithi169 bash[56063]: debug 2023-12-16T07:54:15.358+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:49.497577+0000 front 2023-12-16T07:53:49.497519+0000 (oldest deadline 2023-12-16T07:54:12.897939+0000) 2023-12-16T07:54:16.346 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:16 smithi169 bash[54022]: debug 2023-12-16T07:54:16.018+0000 7fc643ed3700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.017956+0000 front 2023-12-16T07:53:42.017462+0000 (oldest deadline 2023-12-16T07:54:05.517652+0000) 2023-12-16T07:54:16.346 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:16 smithi169 bash[54022]: debug 2023-12-16T07:54:16.018+0000 7fc643ed3700 -1 osd.4 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:48.920271+0000 front 2023-12-16T07:53:48.920086+0000 (oldest deadline 2023-12-16T07:54:13.020176+0000) 2023-12-16T07:54:16.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:16 smithi169 bash[56063]: debug 2023-12-16T07:54:16.338+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:16.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:16 smithi169 bash[56063]: debug 2023-12-16T07:54:16.338+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:49.497577+0000 front 2023-12-16T07:53:49.497519+0000 (oldest deadline 2023-12-16T07:54:12.897939+0000) 2023-12-16T07:54:16.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:16 smithi169 bash[58112]: debug 2023-12-16T07:54:16.370+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:16.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:16 smithi169 bash[58112]: debug 2023-12-16T07:54:16.370+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:16.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:16 smithi169 bash[60136]: debug 2023-12-16T07:54:16.346+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:16.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:16 smithi169 bash[60136]: debug 2023-12-16T07:54:16.346+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:17.686 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:17 smithi169 bash[56063]: debug 2023-12-16T07:54:17.326+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:17.686 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:17 smithi169 bash[56063]: debug 2023-12-16T07:54:17.326+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:49.497577+0000 front 2023-12-16T07:53:49.497519+0000 (oldest deadline 2023-12-16T07:54:12.897939+0000) 2023-12-16T07:54:17.687 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:17 smithi169 bash[58112]: debug 2023-12-16T07:54:17.370+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:17.687 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:17 smithi169 bash[58112]: debug 2023-12-16T07:54:17.370+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:17.687 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:17 smithi169 bash[60136]: debug 2023-12-16T07:54:17.366+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:17.687 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:17 smithi169 bash[60136]: debug 2023-12-16T07:54:17.366+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:18.183 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:17 smithi169 bash[61602]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-4 2023-12-16T07:54:18.183 INFO:journalctl@ceph.osd.4.smithi169.stdout:Dec 16 07:54:18 smithi169 bash[61678]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-4 2023-12-16T07:54:18.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:18 smithi169 bash[56063]: debug 2023-12-16T07:54:18.338+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:18.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:18 smithi169 bash[56063]: debug 2023-12-16T07:54:18.338+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:49.497577+0000 front 2023-12-16T07:53:49.497519+0000 (oldest deadline 2023-12-16T07:54:12.897939+0000) 2023-12-16T07:54:18.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:18 smithi169 bash[58112]: debug 2023-12-16T07:54:18.334+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:18.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:18 smithi169 bash[58112]: debug 2023-12-16T07:54:18.334+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:18.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:18 smithi169 bash[60136]: debug 2023-12-16T07:54:18.350+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:18.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:18 smithi169 bash[60136]: debug 2023-12-16T07:54:18.350+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:19.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:19 smithi169 bash[60136]: debug 2023-12-16T07:54:19.362+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:19.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:19 smithi169 bash[60136]: debug 2023-12-16T07:54:19.362+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:19.684 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:19 smithi169 bash[56063]: debug 2023-12-16T07:54:19.334+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:19.684 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:19 smithi169 bash[56063]: debug 2023-12-16T07:54:19.334+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:49.497577+0000 front 2023-12-16T07:53:49.497519+0000 (oldest deadline 2023-12-16T07:54:12.897939+0000) 2023-12-16T07:54:19.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:19 smithi169 bash[58112]: debug 2023-12-16T07:54:19.318+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:19.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:19 smithi169 bash[58112]: debug 2023-12-16T07:54:19.318+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:20.069 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.4.service' 2023-12-16T07:54:20.105 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T07:54:20.105 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-12-16T07:54:20.106 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-12-16T07:54:20.106 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.5 2023-12-16T07:54:20.315 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:20 smithi169 systemd[1]: Stopping Ceph osd.5 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:54:20.316 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:20 smithi169 bash[56063]: debug 2023-12-16T07:54:20.290+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:20.316 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:20 smithi169 bash[56063]: debug 2023-12-16T07:54:20.290+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:49.497577+0000 front 2023-12-16T07:53:49.497519+0000 (oldest deadline 2023-12-16T07:54:12.897939+0000) 2023-12-16T07:54:20.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:20 smithi169 bash[58112]: debug 2023-12-16T07:54:20.358+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:20.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:20 smithi169 bash[58112]: debug 2023-12-16T07:54:20.358+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:20.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:20 smithi169 bash[60136]: debug 2023-12-16T07:54:20.314+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:20.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:20 smithi169 bash[60136]: debug 2023-12-16T07:54:20.314+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:20.684 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:20 smithi169 bash[56063]: debug 2023-12-16T07:54:20.310+0000 7f0d9f9c9700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:54:20.684 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:20 smithi169 bash[56063]: debug 2023-12-16T07:54:20.310+0000 7f0d9f9c9700 -1 osd.5 143 *** Got signal Terminated *** 2023-12-16T07:54:20.684 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:20 smithi169 bash[56063]: debug 2023-12-16T07:54:20.310+0000 7f0d9f9c9700 -1 osd.5 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:54:21.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:21 smithi169 bash[56063]: debug 2023-12-16T07:54:21.274+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:21.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:21 smithi169 bash[56063]: debug 2023-12-16T07:54:21.274+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:49.497577+0000 front 2023-12-16T07:53:49.497519+0000 (oldest deadline 2023-12-16T07:54:12.897939+0000) 2023-12-16T07:54:21.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:21 smithi169 bash[58112]: debug 2023-12-16T07:54:21.382+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:21.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:21 smithi169 bash[58112]: debug 2023-12-16T07:54:21.382+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:21.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:21 smithi169 bash[60136]: debug 2023-12-16T07:54:21.318+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:21.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:21 smithi169 bash[60136]: debug 2023-12-16T07:54:21.318+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:22.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:22 smithi169 bash[56063]: debug 2023-12-16T07:54:22.318+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:22.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:22 smithi169 bash[56063]: debug 2023-12-16T07:54:22.318+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:49.497577+0000 front 2023-12-16T07:53:49.497519+0000 (oldest deadline 2023-12-16T07:54:12.897939+0000) 2023-12-16T07:54:22.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:22 smithi169 bash[58112]: debug 2023-12-16T07:54:22.398+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:22.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:22 smithi169 bash[58112]: debug 2023-12-16T07:54:22.398+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:22.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:22 smithi169 bash[58112]: debug 2023-12-16T07:54:22.398+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:54:00.011397+0000 front 2023-12-16T07:54:00.011315+0000 (oldest deadline 2023-12-16T07:54:22.311252+0000) 2023-12-16T07:54:22.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:22 smithi169 bash[60136]: debug 2023-12-16T07:54:22.270+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:22.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:22 smithi169 bash[60136]: debug 2023-12-16T07:54:22.270+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:23.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:23 smithi169 bash[56063]: debug 2023-12-16T07:54:23.306+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:23.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:23 smithi169 bash[56063]: debug 2023-12-16T07:54:23.306+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:49.497577+0000 front 2023-12-16T07:53:49.497519+0000 (oldest deadline 2023-12-16T07:54:12.897939+0000) 2023-12-16T07:54:23.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:23 smithi169 bash[56063]: debug 2023-12-16T07:54:23.306+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:57.599296+0000 front 2023-12-16T07:53:57.599367+0000 (oldest deadline 2023-12-16T07:54:22.800024+0000) 2023-12-16T07:54:23.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:23 smithi169 bash[60136]: debug 2023-12-16T07:54:23.306+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:23.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:23 smithi169 bash[60136]: debug 2023-12-16T07:54:23.306+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:23.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:23 smithi169 bash[60136]: debug 2023-12-16T07:54:23.306+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:23.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:23 smithi169 bash[58112]: debug 2023-12-16T07:54:23.398+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:23.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:23 smithi169 bash[58112]: debug 2023-12-16T07:54:23.398+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:23.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:23 smithi169 bash[58112]: debug 2023-12-16T07:54:23.398+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:54:00.011397+0000 front 2023-12-16T07:54:00.011315+0000 (oldest deadline 2023-12-16T07:54:22.311252+0000) 2023-12-16T07:54:24.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:23 smithi007 bash[59950]: level=error ts=2023-12-16T07:54:23.755Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=10 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi169.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:54:24.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:23 smithi007 bash[59950]: level=warn ts=2023-12-16T07:54:23.758Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:54:24.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:23 smithi007 bash[59950]: level=warn ts=2023-12-16T07:54:23.758Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi169.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused" 2023-12-16T07:54:24.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:24 smithi169 bash[56063]: debug 2023-12-16T07:54:24.318+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:38.896288+0000 front 2023-12-16T07:53:38.896402+0000 (oldest deadline 2023-12-16T07:54:09.496494+0000) 2023-12-16T07:54:24.683 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:24 smithi169 bash[56063]: debug 2023-12-16T07:54:24.318+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:49.497577+0000 front 2023-12-16T07:53:49.497519+0000 (oldest deadline 2023-12-16T07:54:12.897939+0000) 2023-12-16T07:54:24.684 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:24 smithi169 bash[56063]: debug 2023-12-16T07:54:24.318+0000 7f0d997bf700 -1 osd.5 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:57.599296+0000 front 2023-12-16T07:53:57.599367+0000 (oldest deadline 2023-12-16T07:54:22.800024+0000) 2023-12-16T07:54:24.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:24 smithi169 bash[58112]: debug 2023-12-16T07:54:24.390+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:24.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:24 smithi169 bash[58112]: debug 2023-12-16T07:54:24.390+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:24.685 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:24 smithi169 bash[58112]: debug 2023-12-16T07:54:24.390+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:54:00.011397+0000 front 2023-12-16T07:54:00.011315+0000 (oldest deadline 2023-12-16T07:54:22.311252+0000) 2023-12-16T07:54:24.685 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:24 smithi169 bash[60136]: debug 2023-12-16T07:54:24.290+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:24.685 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:24 smithi169 bash[60136]: debug 2023-12-16T07:54:24.290+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:24.686 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:24 smithi169 bash[60136]: debug 2023-12-16T07:54:24.290+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:25.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:25 smithi169 bash[58112]: debug 2023-12-16T07:54:25.438+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:25.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:25 smithi169 bash[58112]: debug 2023-12-16T07:54:25.438+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:25.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:25 smithi169 bash[58112]: debug 2023-12-16T07:54:25.438+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:54:00.011397+0000 front 2023-12-16T07:54:00.011315+0000 (oldest deadline 2023-12-16T07:54:22.311252+0000) 2023-12-16T07:54:25.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:25 smithi169 bash[60136]: debug 2023-12-16T07:54:25.318+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:25.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:25 smithi169 bash[60136]: debug 2023-12-16T07:54:25.318+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:25.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:25 smithi169 bash[60136]: debug 2023-12-16T07:54:25.318+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:26.486 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:26 smithi169 bash[61822]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-5 2023-12-16T07:54:26.487 INFO:journalctl@ceph.osd.5.smithi169.stdout:Dec 16 07:54:26 smithi169 bash[61894]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-5 2023-12-16T07:54:26.487 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:26 smithi169 bash[58112]: debug 2023-12-16T07:54:26.414+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:26.487 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:26 smithi169 bash[58112]: debug 2023-12-16T07:54:26.414+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:26.487 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:26 smithi169 bash[58112]: debug 2023-12-16T07:54:26.414+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:54:00.011397+0000 front 2023-12-16T07:54:00.011315+0000 (oldest deadline 2023-12-16T07:54:22.311252+0000) 2023-12-16T07:54:26.487 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:26 smithi169 bash[60136]: debug 2023-12-16T07:54:26.294+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:26.488 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:26 smithi169 bash[60136]: debug 2023-12-16T07:54:26.294+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:26.488 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:26 smithi169 bash[60136]: debug 2023-12-16T07:54:26.294+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:27.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:27 smithi169 bash[58112]: debug 2023-12-16T07:54:27.370+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:27.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:27 smithi169 bash[58112]: debug 2023-12-16T07:54:27.370+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:27.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:27 smithi169 bash[58112]: debug 2023-12-16T07:54:27.370+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:54:00.011397+0000 front 2023-12-16T07:54:00.011315+0000 (oldest deadline 2023-12-16T07:54:22.311252+0000) 2023-12-16T07:54:27.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:27 smithi169 bash[60136]: debug 2023-12-16T07:54:27.342+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:27.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:27 smithi169 bash[60136]: debug 2023-12-16T07:54:27.342+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:27.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:27 smithi169 bash[60136]: debug 2023-12-16T07:54:27.342+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:28.445 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.5.service' 2023-12-16T07:54:28.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T07:54:28.480 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-12-16T07:54:28.480 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-12-16T07:54:28.481 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.6 2023-12-16T07:54:28.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:28 smithi169 bash[60136]: debug 2023-12-16T07:54:28.302+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:28.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:28 smithi169 bash[60136]: debug 2023-12-16T07:54:28.302+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:28.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:28 smithi169 bash[60136]: debug 2023-12-16T07:54:28.302+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:28.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:28 smithi169 bash[58112]: debug 2023-12-16T07:54:28.354+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:28.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:28 smithi169 bash[58112]: debug 2023-12-16T07:54:28.354+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:28.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:28 smithi169 bash[58112]: debug 2023-12-16T07:54:28.354+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:54:00.011397+0000 front 2023-12-16T07:54:00.011315+0000 (oldest deadline 2023-12-16T07:54:22.311252+0000) 2023-12-16T07:54:28.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:28 smithi169 systemd[1]: Stopping Ceph osd.6 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:54:29.183 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:28 smithi169 bash[58112]: debug 2023-12-16T07:54:28.722+0000 7f9298818700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:54:29.183 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:28 smithi169 bash[58112]: debug 2023-12-16T07:54:28.722+0000 7f9298818700 -1 osd.6 143 *** Got signal Terminated *** 2023-12-16T07:54:29.183 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:28 smithi169 bash[58112]: debug 2023-12-16T07:54:28.722+0000 7f9298818700 -1 osd.6 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:54:29.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:29 smithi169 bash[58112]: debug 2023-12-16T07:54:29.326+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:29.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:29 smithi169 bash[58112]: debug 2023-12-16T07:54:29.326+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:29.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:29 smithi169 bash[58112]: debug 2023-12-16T07:54:29.326+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:54:00.011397+0000 front 2023-12-16T07:54:00.011315+0000 (oldest deadline 2023-12-16T07:54:22.311252+0000) 2023-12-16T07:54:29.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:29 smithi169 bash[60136]: debug 2023-12-16T07:54:29.346+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:29.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:29 smithi169 bash[60136]: debug 2023-12-16T07:54:29.346+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:29.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:29 smithi169 bash[60136]: debug 2023-12-16T07:54:29.346+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:30.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:30 smithi169 bash[58112]: debug 2023-12-16T07:54:30.366+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:30.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:30 smithi169 bash[58112]: debug 2023-12-16T07:54:30.366+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:30.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:30 smithi169 bash[58112]: debug 2023-12-16T07:54:30.366+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:54:00.011397+0000 front 2023-12-16T07:54:00.011315+0000 (oldest deadline 2023-12-16T07:54:22.311252+0000) 2023-12-16T07:54:30.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:30 smithi169 bash[60136]: debug 2023-12-16T07:54:30.350+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:30.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:30 smithi169 bash[60136]: debug 2023-12-16T07:54:30.350+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:30.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:30 smithi169 bash[60136]: debug 2023-12-16T07:54:30.350+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:31.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:31 smithi169 bash[60136]: debug 2023-12-16T07:54:31.390+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:31.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:31 smithi169 bash[60136]: debug 2023-12-16T07:54:31.390+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:31.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:31 smithi169 bash[60136]: debug 2023-12-16T07:54:31.390+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:31.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:31 smithi169 bash[58112]: debug 2023-12-16T07:54:31.350+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:31.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:31 smithi169 bash[58112]: debug 2023-12-16T07:54:31.350+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:31.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:31 smithi169 bash[58112]: debug 2023-12-16T07:54:31.350+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:54:00.011397+0000 front 2023-12-16T07:54:00.011315+0000 (oldest deadline 2023-12-16T07:54:22.311252+0000) 2023-12-16T07:54:31.684 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:31 smithi169 bash[58112]: debug 2023-12-16T07:54:31.350+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2023-12-16T07:54:07.513358+0000 front 2023-12-16T07:54:07.513405+0000 (oldest deadline 2023-12-16T07:54:30.913970+0000) 2023-12-16T07:54:32.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:32 smithi169 bash[58112]: debug 2023-12-16T07:54:32.322+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:32.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:32 smithi169 bash[58112]: debug 2023-12-16T07:54:32.322+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:32.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:32 smithi169 bash[58112]: debug 2023-12-16T07:54:32.322+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:54:00.011397+0000 front 2023-12-16T07:54:00.011315+0000 (oldest deadline 2023-12-16T07:54:22.311252+0000) 2023-12-16T07:54:32.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:32 smithi169 bash[58112]: debug 2023-12-16T07:54:32.322+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2023-12-16T07:54:07.513358+0000 front 2023-12-16T07:54:07.513405+0000 (oldest deadline 2023-12-16T07:54:30.913970+0000) 2023-12-16T07:54:32.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:32 smithi169 bash[60136]: debug 2023-12-16T07:54:32.354+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:32.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:32 smithi169 bash[60136]: debug 2023-12-16T07:54:32.354+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:32.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:32 smithi169 bash[60136]: debug 2023-12-16T07:54:32.354+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:32.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:32 smithi169 bash[60136]: debug 2023-12-16T07:54:32.354+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2023-12-16T07:54:03.295276+0000 front 2023-12-16T07:54:03.295315+0000 (oldest deadline 2023-12-16T07:54:32.095812+0000) 2023-12-16T07:54:33.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:33 smithi169 bash[58112]: debug 2023-12-16T07:54:33.294+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:42.006165+0000 front 2023-12-16T07:53:42.006349+0000 (oldest deadline 2023-12-16T07:54:04.807265+0000) 2023-12-16T07:54:33.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:33 smithi169 bash[58112]: debug 2023-12-16T07:54:33.294+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.009675+0000 front 2023-12-16T07:53:50.010010+0000 (oldest deadline 2023-12-16T07:54:15.909715+0000) 2023-12-16T07:54:33.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:33 smithi169 bash[58112]: debug 2023-12-16T07:54:33.294+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:54:00.011397+0000 front 2023-12-16T07:54:00.011315+0000 (oldest deadline 2023-12-16T07:54:22.311252+0000) 2023-12-16T07:54:33.683 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:33 smithi169 bash[58112]: debug 2023-12-16T07:54:33.294+0000 7f9293610700 -1 osd.6 143 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2023-12-16T07:54:07.513358+0000 front 2023-12-16T07:54:07.513405+0000 (oldest deadline 2023-12-16T07:54:30.913970+0000) 2023-12-16T07:54:33.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:33 smithi169 bash[60136]: debug 2023-12-16T07:54:33.362+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:33.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:33 smithi169 bash[60136]: debug 2023-12-16T07:54:33.362+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:33.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:33 smithi169 bash[60136]: debug 2023-12-16T07:54:33.362+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:33.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:33 smithi169 bash[60136]: debug 2023-12-16T07:54:33.362+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2023-12-16T07:54:03.295276+0000 front 2023-12-16T07:54:03.295315+0000 (oldest deadline 2023-12-16T07:54:32.095812+0000) 2023-12-16T07:54:34.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:33 smithi007 bash[59950]: level=error ts=2023-12-16T07:54:33.755Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=10 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi169.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:54:34.069 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:33 smithi007 bash[59950]: level=warn ts=2023-12-16T07:54:33.757Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:54:34.069 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:33 smithi007 bash[59950]: level=warn ts=2023-12-16T07:54:33.758Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi169.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused" 2023-12-16T07:54:34.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:34 smithi169 bash[60136]: debug 2023-12-16T07:54:34.330+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:34.684 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:34 smithi169 bash[60136]: debug 2023-12-16T07:54:34.330+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:34.685 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:34 smithi169 bash[60136]: debug 2023-12-16T07:54:34.330+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:34.685 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:34 smithi169 bash[60136]: debug 2023-12-16T07:54:34.330+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2023-12-16T07:54:03.295276+0000 front 2023-12-16T07:54:03.295315+0000 (oldest deadline 2023-12-16T07:54:32.095812+0000) 2023-12-16T07:54:35.180 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:34 smithi169 bash[62035]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-6 2023-12-16T07:54:35.181 INFO:journalctl@ceph.osd.6.smithi169.stdout:Dec 16 07:54:35 smithi169 bash[62109]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-6 2023-12-16T07:54:35.434 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:35 smithi169 bash[60136]: debug 2023-12-16T07:54:35.354+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:35.434 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:35 smithi169 bash[60136]: debug 2023-12-16T07:54:35.354+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:35.434 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:35 smithi169 bash[60136]: debug 2023-12-16T07:54:35.354+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:35.434 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:35 smithi169 bash[60136]: debug 2023-12-16T07:54:35.354+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2023-12-16T07:54:03.295276+0000 front 2023-12-16T07:54:03.295315+0000 (oldest deadline 2023-12-16T07:54:32.095812+0000) 2023-12-16T07:54:36.433 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:36 smithi169 bash[60136]: debug 2023-12-16T07:54:36.330+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:36.433 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:36 smithi169 bash[60136]: debug 2023-12-16T07:54:36.330+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:36.433 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:36 smithi169 bash[60136]: debug 2023-12-16T07:54:36.330+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:36.434 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:36 smithi169 bash[60136]: debug 2023-12-16T07:54:36.330+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2023-12-16T07:54:03.295276+0000 front 2023-12-16T07:54:03.295315+0000 (oldest deadline 2023-12-16T07:54:32.095812+0000) 2023-12-16T07:54:37.255 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.6.service' 2023-12-16T07:54:37.291 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T07:54:37.291 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-12-16T07:54:37.292 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-12-16T07:54:37.292 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.7 2023-12-16T07:54:37.512 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:37 smithi169 bash[60136]: debug 2023-12-16T07:54:37.326+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:37.513 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:37 smithi169 bash[60136]: debug 2023-12-16T07:54:37.326+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:37.513 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:37 smithi169 bash[60136]: debug 2023-12-16T07:54:37.326+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:37.513 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:37 smithi169 bash[60136]: debug 2023-12-16T07:54:37.326+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2023-12-16T07:54:03.295276+0000 front 2023-12-16T07:54:03.295315+0000 (oldest deadline 2023-12-16T07:54:32.095812+0000) 2023-12-16T07:54:37.513 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:37 smithi169 systemd[1]: Stopping Ceph osd.7 for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:54:37.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:37 smithi169 bash[60136]: debug 2023-12-16T07:54:37.506+0000 7f50678d1700 -1 received signal: Terminated from /sbin/docker-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2023-12-16T07:54:37.933 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:37 smithi169 bash[60136]: debug 2023-12-16T07:54:37.506+0000 7f50678d1700 -1 osd.7 143 *** Got signal Terminated *** 2023-12-16T07:54:37.934 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:37 smithi169 bash[60136]: debug 2023-12-16T07:54:37.506+0000 7f50678d1700 -1 osd.7 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-16T07:54:38.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:38 smithi169 bash[60136]: debug 2023-12-16T07:54:38.338+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:38.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:38 smithi169 bash[60136]: debug 2023-12-16T07:54:38.338+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:38.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:38 smithi169 bash[60136]: debug 2023-12-16T07:54:38.338+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:38.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:38 smithi169 bash[60136]: debug 2023-12-16T07:54:38.338+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2023-12-16T07:54:03.295276+0000 front 2023-12-16T07:54:03.295315+0000 (oldest deadline 2023-12-16T07:54:32.095812+0000) 2023-12-16T07:54:38.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:38 smithi169 bash[60136]: debug 2023-12-16T07:54:38.338+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.169:6804 osd.4 since back 2023-12-16T07:54:12.096478+0000 front 2023-12-16T07:54:12.096698+0000 (oldest deadline 2023-12-16T07:54:37.996487+0000) 2023-12-16T07:54:39.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:39 smithi169 bash[60136]: debug 2023-12-16T07:54:39.386+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:39.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:39 smithi169 bash[60136]: debug 2023-12-16T07:54:39.386+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:39.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:39 smithi169 bash[60136]: debug 2023-12-16T07:54:39.386+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:39.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:39 smithi169 bash[60136]: debug 2023-12-16T07:54:39.386+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2023-12-16T07:54:03.295276+0000 front 2023-12-16T07:54:03.295315+0000 (oldest deadline 2023-12-16T07:54:32.095812+0000) 2023-12-16T07:54:39.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:39 smithi169 bash[60136]: debug 2023-12-16T07:54:39.386+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.169:6804 osd.4 since back 2023-12-16T07:54:12.096478+0000 front 2023-12-16T07:54:12.096698+0000 (oldest deadline 2023-12-16T07:54:37.996487+0000) 2023-12-16T07:54:40.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:40 smithi169 bash[60136]: debug 2023-12-16T07:54:40.362+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:40.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:40 smithi169 bash[60136]: debug 2023-12-16T07:54:40.362+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:40.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:40 smithi169 bash[60136]: debug 2023-12-16T07:54:40.362+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:40.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:40 smithi169 bash[60136]: debug 2023-12-16T07:54:40.362+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2023-12-16T07:54:03.295276+0000 front 2023-12-16T07:54:03.295315+0000 (oldest deadline 2023-12-16T07:54:32.095812+0000) 2023-12-16T07:54:40.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:40 smithi169 bash[60136]: debug 2023-12-16T07:54:40.362+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.169:6804 osd.4 since back 2023-12-16T07:54:12.096478+0000 front 2023-12-16T07:54:12.096698+0000 (oldest deadline 2023-12-16T07:54:37.996487+0000) 2023-12-16T07:54:41.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:41 smithi169 bash[60136]: debug 2023-12-16T07:54:41.322+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:41.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:41 smithi169 bash[60136]: debug 2023-12-16T07:54:41.322+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:41.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:41 smithi169 bash[60136]: debug 2023-12-16T07:54:41.322+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:41.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:41 smithi169 bash[60136]: debug 2023-12-16T07:54:41.322+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2023-12-16T07:54:03.295276+0000 front 2023-12-16T07:54:03.295315+0000 (oldest deadline 2023-12-16T07:54:32.095812+0000) 2023-12-16T07:54:41.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:41 smithi169 bash[60136]: debug 2023-12-16T07:54:41.322+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.169:6804 osd.4 since back 2023-12-16T07:54:12.096478+0000 front 2023-12-16T07:54:12.096698+0000 (oldest deadline 2023-12-16T07:54:37.996487+0000) 2023-12-16T07:54:42.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:42 smithi169 bash[60136]: debug 2023-12-16T07:54:42.310+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2023-12-16T07:53:43.391492+0000 front 2023-12-16T07:53:43.391536+0000 (oldest deadline 2023-12-16T07:54:04.491079+0000) 2023-12-16T07:54:42.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:42 smithi169 bash[60136]: debug 2023-12-16T07:54:42.310+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2023-12-16T07:53:50.392773+0000 front 2023-12-16T07:53:50.393005+0000 (oldest deadline 2023-12-16T07:54:15.692740+0000) 2023-12-16T07:54:42.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:42 smithi169 bash[60136]: debug 2023-12-16T07:54:42.310+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2023-12-16T07:53:58.594144+0000 front 2023-12-16T07:53:58.594525+0000 (oldest deadline 2023-12-16T07:54:23.294320+0000) 2023-12-16T07:54:42.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:42 smithi169 bash[60136]: debug 2023-12-16T07:54:42.310+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2023-12-16T07:54:03.295276+0000 front 2023-12-16T07:54:03.295315+0000 (oldest deadline 2023-12-16T07:54:32.095812+0000) 2023-12-16T07:54:42.683 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:42 smithi169 bash[60136]: debug 2023-12-16T07:54:42.310+0000 7f50626c9700 -1 osd.7 143 heartbeat_check: no reply from 172.21.15.169:6804 osd.4 since back 2023-12-16T07:54:12.096478+0000 front 2023-12-16T07:54:12.096698+0000 (oldest deadline 2023-12-16T07:54:37.996487+0000) 2023-12-16T07:54:43.737 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:43 smithi169 bash[62255]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-7 2023-12-16T07:54:43.737 INFO:journalctl@ceph.osd.7.smithi169.stdout:Dec 16 07:54:43 smithi169 bash[62331]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-osd-7 2023-12-16T07:54:44.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:43 smithi007 bash[59950]: level=error ts=2023-12-16T07:54:43.756Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=10 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi169.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused" 2023-12-16T07:54:44.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:43 smithi007 bash[59950]: level=warn ts=2023-12-16T07:54:43.758Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:54:44.068 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:43 smithi007 bash[59950]: level=warn ts=2023-12-16T07:54:43.760Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi169.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused" 2023-12-16T07:54:45.686 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@osd.7.service' 2023-12-16T07:54:45.721 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T07:54:45.722 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-12-16T07:54:45.722 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2023-12-16T07:54:45.722 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@prometheus.a 2023-12-16T07:54:45.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:45 smithi169 systemd[1]: Stopping Ceph prometheus.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:54:45.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:45 smithi169 bash[49087]: ts=2023-12-16T07:54:45.827Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2023-12-16T07:54:45.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:45 smithi169 bash[49087]: ts=2023-12-16T07:54:45.827Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2023-12-16T07:54:45.933 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:45 smithi169 bash[49087]: ts=2023-12-16T07:54:45.827Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2023-12-16T07:54:45.934 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:45 smithi169 bash[49087]: ts=2023-12-16T07:54:45.827Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2023-12-16T07:54:45.934 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:45 smithi169 bash[49087]: ts=2023-12-16T07:54:45.827Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2023-12-16T07:54:45.934 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:45 smithi169 bash[49087]: ts=2023-12-16T07:54:45.827Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2023-12-16T07:54:45.934 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:45 smithi169 bash[49087]: ts=2023-12-16T07:54:45.828Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2023-12-16T07:54:45.934 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:45 smithi169 bash[49087]: ts=2023-12-16T07:54:45.828Z caller=main.go:828 level=info msg="Scrape manager stopped" 2023-12-16T07:54:45.934 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:45 smithi169 bash[49087]: ts=2023-12-16T07:54:45.828Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2023-12-16T07:54:45.934 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:45 smithi169 bash[49087]: ts=2023-12-16T07:54:45.828Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2023-12-16T07:54:45.934 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:45 smithi169 bash[49087]: ts=2023-12-16T07:54:45.828Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2023-12-16T07:54:45.934 INFO:journalctl@ceph.prometheus.a.smithi169.stdout:Dec 16 07:54:45 smithi169 bash[49087]: ts=2023-12-16T07:54:45.829Z caller=main.go:1066 level=info msg="See you next time!" 2023-12-16T07:54:46.546 DEBUG:teuthology.orchestra.run.smithi169:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@prometheus.a.service' 2023-12-16T07:54:46.581 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-16T07:54:46.581 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2023-12-16T07:54:46.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 --force --keep-logs 2023-12-16T07:54:51.068 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:54:50 smithi007 systemd[1]: Stopping Ceph node-exporter.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:54:51.495 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:54:51 smithi007 bash[80100]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-node-exporter-a 2023-12-16T07:54:51.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:51 smithi007 bash[59950]: level=warn ts=2023-12-16T07:54:51.624Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2023-12-16T07:54:51.818 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:51 smithi007 bash[59950]: level=warn ts=2023-12-16T07:54:51.626Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi169.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.169:8443: connect: connection refused" 2023-12-16T07:54:51.818 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:54:51 smithi007 bash[80148]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-node-exporter.a 2023-12-16T07:54:51.818 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:54:51 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2023-12-16T07:54:51.818 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:54:51 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2023-12-16T07:54:51.818 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Dec 16 07:54:51 smithi007 systemd[1]: Stopped Ceph node-exporter.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:54:52.174 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:51 smithi007 systemd[1]: Stopping Ceph alertmanager.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:54:52.174 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:51 smithi007 bash[59950]: level=info ts=2023-12-16T07:54:51.953Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2023-12-16T07:54:52.817 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:52 smithi007 bash[80186]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-alertmanager-a 2023-12-16T07:54:52.818 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:52 smithi007 bash[80237]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-alertmanager.a 2023-12-16T07:54:52.818 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:52 smithi007 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@alertmanager.a.service: Succeeded. 2023-12-16T07:54:52.818 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Dec 16 07:54:52 smithi007 systemd[1]: Stopped Ceph alertmanager.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:55:26.521 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 --force --keep-logs 2023-12-16T07:55:30.933 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:55:30 smithi169 systemd[1]: Stopping Ceph node-exporter.b for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:55:31.908 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:55:31 smithi169 bash[62716]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-node-exporter-b 2023-12-16T07:55:31.908 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:55:31 smithi169 bash[62763]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-node-exporter.b 2023-12-16T07:55:31.909 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:55:31 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2023-12-16T07:55:31.909 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:55:31 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2023-12-16T07:55:31.909 INFO:journalctl@ceph.node-exporter.b.smithi169.stdout:Dec 16 07:55:31 smithi169 systemd[1]: Stopped Ceph node-exporter.b for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:55:32.183 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:55:31 smithi169 systemd[1]: Stopping Ceph grafana.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715... 2023-12-16T07:55:32.183 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:55:31 smithi169 bash[47688]: t=2023-12-16T07:55:31+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2023-12-16T07:55:32.483 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:55:32 smithi169 bash[62800]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-grafana-a 2023-12-16T07:55:32.901 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:55:32 smithi169 bash[62846]: Error response from daemon: No such container: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715-grafana.a 2023-12-16T07:55:32.902 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:55:32 smithi169 systemd[1]: ceph-8b71d3dc-9be4-11ee-95a5-87774f69a715@grafana.a.service: Succeeded. 2023-12-16T07:55:32.902 INFO:journalctl@ceph.grafana.a.smithi169.stdout:Dec 16 07:55:32 smithi169 systemd[1]: Stopped Ceph grafana.a for 8b71d3dc-9be4-11ee-95a5-87774f69a715. 2023-12-16T07:55:55.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-12-16T07:55:55.502 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-12-16T07:55:55.515 INFO:tasks.cephadm:Archiving crash dumps... 2023-12-16T07:55:55.518 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi/7493743/remote/smithi007/crash 2023-12-16T07:55:55.518 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/crash -- . 2023-12-16T07:55:55.531 INFO:teuthology.orchestra.run.smithi007.stderr:tar: /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/crash: Cannot open: No such file or directory 2023-12-16T07:55:55.532 INFO:teuthology.orchestra.run.smithi007.stderr:tar: Error is not recoverable: exiting now 2023-12-16T07:55:55.533 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi/7493743/remote/smithi169/crash 2023-12-16T07:55:55.534 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/crash -- . 2023-12-16T07:55:55.549 INFO:teuthology.orchestra.run.smithi169.stderr:tar: /var/lib/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/crash: Cannot open: No such file or directory 2023-12-16T07:55:55.549 INFO:teuthology.orchestra.run.smithi169.stderr:tar: Error is not recoverable: exiting now 2023-12-16T07:55:55.550 INFO:tasks.cephadm:Checking cluster log for badness... 2023-12-16T07:55:55.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-12-16T07:55:55.592 INFO:teuthology.orchestra.run.smithi007.stderr:grep: /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph.log: No such file or directory 2023-12-16T07:55:55.594 INFO:tasks.cephadm:Compressing logs... 2023-12-16T07:55:55.595 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-16T07:55:55.597 DEBUG:teuthology.orchestra.run.smithi169:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-16T07:55:55.609 INFO:teuthology.orchestra.run.smithi007.stderr:find: '/var/log/rbd-target-api': No such file or directory 2023-12-16T07:55:55.614 INFO:teuthology.orchestra.run.smithi169.stderr:find: '/var/log/rbd-target-api': No such file or directory 2023-12-16T07:55:55.618 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-16T07:55:55.618 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mon.a.log 2023-12-16T07:55:55.619 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mgr.y.log 2023-12-16T07:55:55.620 INFO:teuthology.orchestra.run.smithi007.stderr:gzip /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mon.a.log: -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mon.c.log 2023-12-16T07:55:55.620 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-client.rgw.foo.smithi007.mglnhl.log 2023-12-16T07:55:55.621 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mgr.y.log: /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.1.log 2023-12-16T07:55:55.621 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-client.rgw.foo.smithi007.mglnhl.log: gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/tcmu-runner.log 2023-12-16T07:55:55.622 INFO:teuthology.orchestra.run.smithi007.stderr: 62.7% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-client.rgw.foo.smithi007.mglnhl.log.gz 2023-12-16T07:55:55.622 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.0.log 2023-12-16T07:55:55.623 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.1.log: gzip /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/tcmu-runner.log: -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-client.rgw.smpl.smithi007.tizuzt.log 2023-12-16T07:55:55.623 INFO:teuthology.orchestra.run.smithi007.stderr: 82.5% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/tcmu-runner.log.gz 2023-12-16T07:55:55.625 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.3.log 2023-12-16T07:55:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-16T07:55:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mon.b.log 2023-12-16T07:55:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-client.rgw.smpl.smithi169.aiyfmr.log 2023-12-16T07:55:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.5.log 2023-12-16T07:55:55.626 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mgr.x.log 2023-12-16T07:55:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-client.rgw.smpl.smithi169.aiyfmr.log: 62.5%/var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.5.log: -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-client.rgw.smpl.smithi169.aiyfmr.log.gz 2023-12-16T07:55:55.627 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.6.log 2023-12-16T07:55:55.629 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-client.rgw.foo.smithi169.okfkhg.log 2023-12-16T07:55:55.630 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-client.rgw.smpl.smithi007.tizuzt.log: gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-volume.log 2023-12-16T07:55:55.630 INFO:teuthology.orchestra.run.smithi007.stderr: 62.7% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-client.rgw.smpl.smithi007.tizuzt.log.gz 2023-12-16T07:55:55.630 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.2.log 2023-12-16T07:55:55.632 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mgr.x.log: /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.4.log 2023-12-16T07:55:55.632 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-client.rgw.foo.smithi169.okfkhg.log: 62.5% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-client.rgw.foo.smithi169.okfkhg.log.gz 2023-12-16T07:55:55.632 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-volume.log 2023-12-16T07:55:55.633 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.7.log 2023-12-16T07:55:55.637 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-volume.log: 91.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-16T07:55:55.643 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.4.log: /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-volume.log: /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.7.log: 86.7% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-16T07:55:55.644 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.2.log: 91.2% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-volume.log.gz 2023-12-16T07:55:55.645 INFO:teuthology.orchestra.run.smithi169.stderr: 91.3% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-volume.log.gz 2023-12-16T07:55:55.679 INFO:teuthology.orchestra.run.smithi169.stderr: 89.2% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mgr.x.log.gz 2023-12-16T07:55:55.954 INFO:teuthology.orchestra.run.smithi169.stderr: 92.6% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mon.b.log.gz 2023-12-16T07:55:55.999 INFO:teuthology.orchestra.run.smithi007.stderr: 88.8% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mgr.y.log.gz 2023-12-16T07:55:56.067 INFO:teuthology.orchestra.run.smithi007.stderr: 92.6% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mon.c.log.gz 2023-12-16T07:55:56.562 INFO:teuthology.orchestra.run.smithi169.stderr: 93.2% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.6.log.gz 2023-12-16T07:55:56.610 INFO:teuthology.orchestra.run.smithi007.stderr: 93.2% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.2.log.gz 2023-12-16T07:55:56.735 INFO:teuthology.orchestra.run.smithi007.stderr: 93.6% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.0.log.gz 2023-12-16T07:55:56.813 INFO:teuthology.orchestra.run.smithi169.stderr: 93.7% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.7.log.gz 2023-12-16T07:55:56.814 INFO:teuthology.orchestra.run.smithi007.stderr: 91.1% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-mon.a.log.gz 2023-12-16T07:55:56.930 INFO:teuthology.orchestra.run.smithi169.stderr: 93.4% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.5.log.gz 2023-12-16T07:55:57.033 INFO:teuthology.orchestra.run.smithi169.stderr: 93.7% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.4.log.gz 2023-12-16T07:55:57.034 INFO:teuthology.orchestra.run.smithi169.stderr: 2023-12-16T07:55:57.034 INFO:teuthology.orchestra.run.smithi169.stderr:real 0m1.432s 2023-12-16T07:55:57.034 INFO:teuthology.orchestra.run.smithi169.stderr:user 0m5.077s 2023-12-16T07:55:57.034 INFO:teuthology.orchestra.run.smithi169.stderr:sys 0m0.189s 2023-12-16T07:55:57.119 INFO:teuthology.orchestra.run.smithi007.stderr: 93.6% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.1.log.gz 2023-12-16T07:55:57.121 INFO:teuthology.orchestra.run.smithi007.stderr: 93.6% -- replaced with /var/log/ceph/8b71d3dc-9be4-11ee-95a5-87774f69a715/ceph-osd.3.log.gz 2023-12-16T07:55:57.123 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-12-16T07:55:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:real 0m1.522s 2023-12-16T07:55:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:user 0m6.917s 2023-12-16T07:55:57.123 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m0.237s 2023-12-16T07:55:57.123 INFO:tasks.cephadm:Archiving logs... 2023-12-16T07:55:57.124 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi/7493743/remote/smithi007/log 2023-12-16T07:55:57.125 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-16T07:55:58.042 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi/7493743/remote/smithi169/log 2023-12-16T07:55:58.043 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-16T07:55:58.814 INFO:tasks.cephadm:Removing cluster... 2023-12-16T07:55:58.814 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 --force 2023-12-16T07:55:59.611 DEBUG:teuthology.orchestra.run.smithi169:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 --force 2023-12-16T07:56:00.405 INFO:tasks.cephadm:Removing cephadm ... 2023-12-16T07:56:00.406 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-16T07:56:00.413 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-16T07:56:00.419 INFO:tasks.cephadm:Teardown complete 2023-12-16T07:56:00.419 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-16T07:56:00.429 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-16T07:56:00.429 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T07:56:00.432 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-16T07:56:00.451 INFO:teuthology.orchestra.run.smithi007.stdout: remote refid st t when poll reach delay offset jitter 2023-12-16T07:56:00.451 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================== 2023-12-16T07:56:00.451 INFO:teuthology.orchestra.run.smithi007.stdout:*hv01.front.sepi 44.4.53.2 2 u 15 64 377 0.089 3.142 0.734 2023-12-16T07:56:00.451 INFO:teuthology.orchestra.run.smithi007.stdout:+hv02.front.sepi 63.231.80.2 3 u 12 64 377 0.091 3.739 0.352 2023-12-16T07:56:00.451 INFO:teuthology.orchestra.run.smithi007.stdout:+hv03.front.sepi 74.6.168.72 3 u 7 64 377 0.087 2.215 0.513 2023-12-16T07:56:00.452 INFO:teuthology.orchestra.run.smithi007.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-12-16T07:56:00.453 INFO:teuthology.orchestra.run.smithi169.stdout: remote refid st t when poll reach delay offset jitter 2023-12-16T07:56:00.454 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================== 2023-12-16T07:56:00.454 INFO:teuthology.orchestra.run.smithi169.stdout:*hv01.front.sepi 44.4.53.2 2 u 19 128 377 0.078 0.156 0.088 2023-12-16T07:56:00.454 INFO:teuthology.orchestra.run.smithi169.stdout:+hv02.front.sepi 63.231.80.2 3 u 18 64 377 0.108 1.172 0.167 2023-12-16T07:56:00.454 INFO:teuthology.orchestra.run.smithi169.stdout:+hv03.front.sepi 74.6.168.72 3 u 17 128 377 0.084 0.197 0.135 2023-12-16T07:56:00.454 INFO:teuthology.orchestra.run.smithi169.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2023-12-16T07:56:00.455 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-16T07:56:00.464 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-16T07:56:00.464 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-16T07:56:00.545 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-16T07:56:00.554 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-16T07:56:00.563 INFO:teuthology.task.internal:Duration was 2098.170843 seconds 2023-12-16T07:56:00.563 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-16T07:56:00.573 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-16T07:56:00.573 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-16T07:56:00.576 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-16T07:56:00.630 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-16T07:56:00.631 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2023-12-16T07:56:00.631 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-16T07:56:00.645 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2023-12-16T07:56:00.645 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-16T07:56:00.658 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-16T07:56:00.659 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-16T07:56:00.661 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-16T07:56:00.758 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-16T07:56:00.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-16T07:56:00.761 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-16T07:56:00.922 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-16T07:56:00.932 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-16T07:56:00.932 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-16T07:56:00.948 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-16T07:56:00.963 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-16T07:56:00.973 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-16T07:56:00.994 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-16T07:56:01.005 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2023-12-16T07:56:01.020 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2023-12-16T07:56:01.053 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-16T07:56:01.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T07:56:01.059 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-16T07:56:01.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-16T07:56:01.067 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-16T07:56:01.080 INFO:teuthology.task.internal:Transferring archived files... 2023-12-16T07:56:01.081 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi/7493743/remote/smithi007 2023-12-16T07:56:01.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-16T07:56:01.137 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi/7493743/remote/smithi169 2023-12-16T07:56:01.138 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-16T07:56:01.168 INFO:teuthology.task.internal:Removing archive directory... 2023-12-16T07:56:01.168 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-16T07:56:01.171 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-16T07:56:01.176 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-16T07:56:01.190 INFO:teuthology.task.internal:Not uploading archives. 2023-12-16T07:56:01.190 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-16T07:56:01.200 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-16T07:56:01.201 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-16T07:56:01.203 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-16T07:56:01.207 INFO:teuthology.orchestra.run.smithi007.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 16 07:56 /home/ubuntu/cephtest 2023-12-16T07:56:01.210 INFO:teuthology.orchestra.run.smithi169.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 16 07:56 /home/ubuntu/cephtest 2023-12-16T07:56:01.213 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-16T07:56:01.223 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-16T07:56:01.243 INFO:teuthology.nuke:Checking targets against current locks 2023-12-16T07:56:01.272 DEBUG:teuthology.nuke:shortname: smithi007 2023-12-16T07:56:01.272 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T07:56:01.295 DEBUG:teuthology.nuke:shortname: smithi169 2023-12-16T07:56:01.295 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-16T07:56:01.298 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi/7493743', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 07:11:03.918700', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-12-16T07:56:01.320 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-16_00:10:43-orch-main-distro-default-smithi/7493743', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-16 07:11:03.920678', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='} 2023-12-16T07:56:01.350 INFO:teuthology.orchestra.console.smithi007:Power off 2023-12-16T07:56:01.351 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-16T07:56:01.380 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2023-12-16T07:56:01.380 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T07:56:01.412 INFO:teuthology.orchestra.console.smithi169:Power off 2023-12-16T07:56:01.413 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-16T07:56:01.426 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2023-12-16T07:56:01.440 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2023-12-16T07:56:01.440 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T07:56:01.464 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2023-12-16T07:56:05.426 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T07:56:05.465 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T07:56:05.542 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2023-12-16T07:56:05.577 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2023-12-16T07:56:09.545 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T07:56:09.578 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T07:56:09.657 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2023-12-16T07:56:09.692 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2023-12-16T07:56:13.661 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T07:56:13.693 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-16T07:56:13.775 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2023-12-16T07:56:13.805 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2023-12-16T07:56:13.877 INFO:teuthology.orchestra.console.smithi007:Power off completed 2023-12-16T07:56:13.906 INFO:teuthology.orchestra.console.smithi169:Power off completed 2023-12-16T07:56:14.011 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} duration: 2098.170842885971 failure_reason: 'Command failed on smithi007 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8b71d3dc-9be4-11ee-95a5-87774f69a715 -e sha1=6d933094a156ec7587742e5aa25fba5ba27ec678 -- bash -c \''ceph versions | jq -e \''"\''"\''.overall | length == 1\''"\''"\''\''''' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8136ccd4f2ae4b8aa737eb1184bdf6da status: fail success: false 2023-12-16T07:56:14.012 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-16T07:56:14.104 INFO:teuthology.run:FAIL